o +ifh@sdZddlZddlZddlZddlZddlZddlZddlmZddlm Z ddl m Z m Z m Z mZmZddlZddlmZddlmZmZdd lmZdd lmZmZdd lmZdd lmZmZdd l m!Z!ddl"m#Z#ddl$m%Z%m&Z&e#e'Z(eGdddZ)eGdddZ*Gddde+Z,Gddde+Z-eGdddZ.eGdddZ/Gddde e0e/fZ1eGdddZ2dS) aDatasetInfo and MetricInfo record information we know about a dataset and a metric. This includes things that we know about the dataset statically, i.e.: - description - canonical location - does it have validation and tests splits - size - etc. This also includes the things that can and should be computed once we've processed the dataset as well: - number of examples (in each split) - etc. N) dataclass)Path)ClassVarDictListOptionalUnion) url_to_fs) DatasetCardDatasetCardData)config)FeaturesValue) SplitDict) TaskTemplatetask_template_from_dict)Version) get_logger)asdict unique_valuesc@&eZdZUdZeed<dZeed<dS)SupervisedKeysDatainputoutputN)__name__ __module__ __qualname__rstr__annotations__rr!r!F/var/www/html/corbot_env/lib/python3.10/site-packages/datasets/info.pyr9 rc@r)DownloadChecksumsEntryDatarkeyvalueN)rrrr%rr r&r!r!r!r"r$?r#r$c@eZdZdZdS)MissingCachedSizesConfigErrorz;The expected cached sizes of the download file are missing.Nrrr__doc__r!r!r!r"r(Er(c@r')NonMatchingCachedSizesErrorz/The prepared split doesn't have expected sizes.Nr)r!r!r!r"r,Ir+r,c@sLeZdZUdZeeed<dZeeed<ddZ e deddfdd Z dS) PostProcessedInfoNfeaturesresources_checksumscCs0|jdurt|jtst|j|_dSdSdSN)r. isinstancer from_dictselfr!r!r" __post_init__RszPostProcessedInfo.__post_init__post_processed_info_dictreturnc4ddt|D|difdd|DS)NcSh|]}|jqSr!name.0fr!r!r" Yz.PostProcessedInfo.from_dict..ci|] \}}|vr||qSr!r!r=kv field_namesr!r" Zz/PostProcessedInfo.from_dict..r! dataclassesfieldsitems)clsr6r!rEr"r2W zPostProcessedInfo.from_dict) rrrr.rrr r/dictr5 classmethodr2r!r!r!r"r-Ms r-c@seZdZUdZejedZeed<ejedZ eed<ejedZ eed<ejedZ eed<dZ e eed<dZe eed <dZe eed <dZe eeed <dZe eed <dZe eed <dZe eed<dZe eeefed<dZe eed<dZe eed<dZe e ed<dZ!e e ed<dZ"e e ed<dZ#e e ed<gdZ$e%eeed<ddZ& d8de efddZ'd9dd Z(d!d"Z)e*d#edfd$d%Z+e* d:d&ede ed'dfd(d)Z,e*d*ed'dfd+d,Z-d;d DatasetInfoa Information about a dataset. `DatasetInfo` documents datasets, including its name, version, and features. See the constructor arguments and properties for a full list. Not all fields are known on construction and may be updated later. Attributes: description (`str`): A description of the dataset. citation (`str`): A BibTeX citation of the dataset. homepage (`str`): A URL to the official homepage for the dataset. license (`str`): The dataset's license. It can be the name of the license or a paragraph containing the terms of the license. features ([`Features`], *optional*): The features used to specify the dataset's column types. post_processed (`PostProcessedInfo`, *optional*): Information regarding the resources of a possible post-processing of a dataset. For example, it can contain the information of an index. supervised_keys (`SupervisedKeysData`, *optional*): Specifies the input feature and the label for supervised learning if applicable for the dataset (legacy from TFDS). builder_name (`str`, *optional*): The name of the `GeneratorBasedBuilder` subclass used to create the dataset. Usually matched to the corresponding script name. It is also the snake_case version of the dataset builder class name. config_name (`str`, *optional*): The name of the configuration derived from [`BuilderConfig`]. version (`str` or [`Version`], *optional*): The version of the dataset. splits (`dict`, *optional*): The mapping between split name and metadata. download_checksums (`dict`, *optional*): The mapping between the URL to download the dataset's checksums and corresponding metadata. download_size (`int`, *optional*): The size of the files to download to generate the dataset, in bytes. post_processing_size (`int`, *optional*): Size of the dataset in bytes after post-processing, if any. dataset_size (`int`, *optional*): The combined size in bytes of the Arrow tables for all splits. size_in_bytes (`int`, *optional*): The combined size in bytes of all files associated with the dataset (downloaded files + Arrow files). task_templates (`List[TaskTemplate]`, *optional*): The task templates to prepare the dataset for during training and evaluation. Each template casts the dataset's [`Features`] to standardized column names and types as detailed in `datasets.tasks`. **config_kwargs (additional keyword arguments): Keyword arguments to be passed to the [`BuilderConfig`] and used in the [`DatasetBuilder`]. default_factory descriptioncitationhomepagelicenseNr.post_processedsupervised_keystask_templates builder_name dataset_name config_nameversionsplitsdownload_checksums download_sizepost_processing_size dataset_size size_in_bytes)r]rarcr.r__INCLUDED_INFO_IN_YAMLcsjdurtjtstj_jdur$tjts$tj_jdurCtjtsCtjtrsz-DatasetInfo.__post_init__..cSsg|]}|dur|qSr0r!rfr!r!r"rhscsg|]}|jqSr!)align_with_featuresr.rfr3r!r"rhs r!)r.r1rr2rXr-r^rrr_rfrom_split_dictrYrtuplelistrZrr)r4 templatesrgr!r3r"r5s@          zDatasetInfo.__post_init__F deprecatedstorage_optionscCs|dkr tdt|j}t|fi|pi^}}|t|tj d}|j ||dWdn1s6wY|j ra|t|tj d}| |WddS1sZwYdSdS)ahWrite `DatasetInfo` and license (if present) as JSON files to `dataset_info_dir`. Args: dataset_info_dir (`str`): Destination directory. pretty_print (`bool`, defaults to `False`): If `True`, the JSON will be pretty-printed with the indent level of 4. fs (`fsspec.spec.AbstractFileSystem`, *optional*): Instance of the remote filesystem used to download the files from. `fs` was deprecated in version 2.9.0 and will be removed in 3.0.0. Please use `storage_options` instead, e.g. `storage_options=fs.storage_options`. storage_options (`dict`, *optional*): Key/value pairs to be passed on to the file-system backend, if any. Example: ```py >>> from datasets import load_dataset >>> ds = load_dataset("rotten_tomatoes", split="validation") >>> ds.info.write_to_directory("/path/to/directory/") ``` rn'fs' was deprecated in favor of 'storage_options' in version 2.9.0 and will be removed in 3.0.0. You can remove this warning by passing 'storage_options=fs.storage_options' instead.wb) pretty_printN)warningswarn FutureWarningror open posixpathjoinr DATASET_INFO_FILENAME _dump_inforWLICENSE_FILENAME _dump_license)r4dataset_info_dirrrfsro_r>r!r!r"write_to_directorys! "zDatasetInfo.write_to_directorycCs*|tjt||r dnddddS)zQDump info in `file` file-like object open in bytes mode (to support remote files)Nindentutf-8)writejsondumpsrencode)r4filerrr!r!r"rzs*zDatasetInfo._dump_infocCs||jddS)zTDump license in `file` file-like object open in bytes mode (to support remote files)rN)rrWr)r4rr!r!r"r| szDatasetInfo._dump_license dataset_infosc s(ddDtdkrtfddDrdSdtddD}dtddD}dtd dD}dtd dD}d}d}d}d dD} t| d krwtt| dj| d d}n t| rtt| d}|r|nd}||||||||d S)NcSsg|] }|dur|qSr0)copyr= dset_infor!r!r"rhz*DatasetInfo.from_merge..rc3s|] }d|kVqdS)rNr!rrr!r" sz)DatasetInfo.from_merge..z cs|]}|jVqdSr0)rTr=infor!r!r"rcsrr0)rUrr!r!r"rrcsrr0)rVrr!r!r"rrcsrr0)rWrr!r!r"rrcSsg|] }|jdur|jqSr0)rZrr!r!r"rhrr )rTrUrVrWr.rYrZ)lenallrxrstriprlset intersection) rMrrTrUrVrWr.rYrZall_task_templatesr!rr" from_merges2"   zDatasetInfo.from_merger}r7cCs|dkr tdt|j}t|fi|pi^}}td||s'td|jt |t j ddd }t |}Wdn1sDwY||S) aCreate [`DatasetInfo`] from the JSON file in `dataset_info_dir`. This function updates all the dynamically generated fields (num_examples, hash, time of creation,...) of the [`DatasetInfo`]. This will overwrite all previous metadata. Args: dataset_info_dir (`str`): The directory containing the metadata file. This should be the root directory of a specific dataset version. fs (`fsspec.spec.AbstractFileSystem`, *optional*): Instance of the remote filesystem used to download the files from. `fs` was deprecated in version 2.9.0 and will be removed in 3.0.0. Please use `storage_options` instead, e.g. `storage_options=fs.storage_options`. storage_options (`dict`, *optional*): Key/value pairs to be passed on to the file-system backend, if any. Example: ```py >>> from datasets import DatasetInfo >>> ds_info = DatasetInfo.from_directory("/path/to/directory/") ``` rnrpzLoading Dataset info from zECalling DatasetInfo.from_directory() with undefined dataset_info_dir.rrencodingN)rsrtruror loggerr ValueErrorrvrwrxr ryrloadr2)rMr}r~rorr>dataset_info_dictr!r!r"from_directory1s%  zDatasetInfo.from_directoryrcr8)NcSr9r!r:r<r!r!r"r?ir@z(DatasetInfo.from_dict..crAr!r!rBrEr!r"rGjrHz)DatasetInfo.from_dict..r!rI)rMrr!rEr"r2grNzDatasetInfo.from_dictTother_dataset_infocs.|j}|jdifdd|jDdS)Ncs(i|]\}}|dus s|t|qSr0rdeepcopyrB ignore_noner!r"rGos   z&DatasetInfo.update..r!)__dict__updaterL)r4rr self_dictr!rr"rls   zDatasetInfo.updatecCs |jdidd|jDS)NcSi|] \}}|t|qSr!rrBr!r!r"rGwrz$DatasetInfo.copy..r!) __class__rrLr3r!r!r"rvs zDatasetInfo.copycCsfi}t|}|D](}||jvr0t||}t|dr |||<qt|dr,|||<q|||<q|S)N _to_yaml_list_to_yaml_string)rregetattrhasattrrr)r4 yaml_dictrr%r&r!r!r" _to_yaml_dictys    zDatasetInfo._to_yaml_dict yaml_datacs~t|}|ddurt|d|d<|ddur%t|d|d<ddt|D|difdd|DS)Nr.r_cSr9r!r:r<r!r!r"r?r@z.DatasetInfo._from_yaml_dict..crAr!r!rBrEr!r"rGrHz/DatasetInfo._from_yaml_dict..r!) rrgetr_from_yaml_listrrJrKrL)rMrr!rEr"_from_yaml_dicts  zDatasetInfo._from_yaml_dict)FrnNF)rnN)T)rrQ)r7rQ)2rrrr*rJfieldrrTr rUrVrWr.rrrXr-rYrrZrrr[r\r]r^rrr_rOr`raintrbrcrdrerr5rrzr|rPrrr2rrrrr!r!r!r"rQ]s\ /* 1" 5  rQc@sNeZdZd dddZedddZededdfd d Zdeddfd d ZdS)DatasetInfosDictFr7Nc Cs i}tj|tj}tj|tj}|s||}||tj|rPt |ddd}dd| D}t j |||r=dnddWdn1sKwYtj|r_t |} | j} nd} t} |r|| | duryt dt| d n| } | t|dSdS) NwrrcSsi|] \}}|t|qSr!)rr=r]rr!r!r"rGsz7DatasetInfosDict.write_to_directory..rrz--- z --- )ospathrxr DATASETDICT_INFOS_FILENAMEREPOCARD_FILENAMErrexistsrvrLrdumpr rdatar to_dataset_card_datarsaver) r4dataset_infos_dir overwriterrtotal_dataset_infosdataset_infos_pathdataset_readme_pathr>dataset_infos_dict dataset_carddataset_card_datar!r!r"rs0      z#DatasetInfosDict.write_to_directorycCstd|tjtj|tjr(t t |tjj }d|vr(| |Stjtj|tj r`ttj|tj dd}|ddt |DWdS1sYwYdS|S)NzLoading Dataset Infos from dataset_inforrcSrr!)rQr2)r=r]rr!r!r"rGs z3DatasetInfosDict.from_directory..)rrrrrrxr rr rrrfrom_dataset_card_datarrvrrL)rMrrr>r!r!r"rs  $zDatasetInfosDict.from_directoryrcCslt|dttfr3t|dtr|dd|dDSt|d}|ddd|_||j|iS|S)NrcSs i|] }|ddt|qS)r]default)rrQr)r=dataset_info_yaml_dictr!r!r"rGs z;DatasetInfosDict.from_dataset_card_data..r]r)r1rrlrOrQrr])rMrrr!r!r"rs z'DatasetInfosDict.from_dataset_card_datacCs6|rd|vrt|dtr|ddd|di}nd|vr/t|dtr/dd|dD}ni}i|dd|D}|D]\}}||d<qBt|dkrutt||d<|d dd}|dkrsd|i|d|d<dSdSg|d<t |D]\}}| ddd|i|}|d |qdSdS)Nrr]rcSsi|]}|d|qS)r]r!)r=config_metadatar!r!r"rGsz9DatasetInfosDict.to_dataset_card_data..cSsi|] \}}||qSr!)rrr!r!r"rGsr ) r1rOrrlrLrnextitervaluespopsortedappend)r4rdataset_metadata_infosrr]dset_info_yaml_dictrr!r!r"rs@     !z%DatasetInfosDict.to_dataset_card_data)FF)r7N)r7r) rrrrrPrr rrr!r!r!r"rs  rc@seZdZUdZeed<eed<eed<ejedZ eed<ejedZ eed<ejedZ eed<eje dZ eeed <eje dZeeed <d Zeed <d Zeeed<d Zeeed<d Zeeed<d Zeeed<ddZdddZedddZededdfddZd S) MetricInfoaInformation about a metric. `MetricInfo` documents a metric, including its name, version, and features. See the constructor arguments and properties for a full list. Note: Not all fields are known on construction and may be updated later. rTrUr.rRinputs_descriptionrVrW codebase_urlsreference_urlsF streamableNformat metric_namer] experiment_idcCsH|jdur |jD]\}}t|tstd|d|jjq dSdS)NzSWhen using 'numpy' format, all features should be a `datasets.Value` feature. Here z is an instance of )rr.rLr1rrrr)r4r%r&r!r!r"r5s  zMetricInfo.__post_init__cCsttj|tjddd}tjt|||rdnddWdn1s&wY|j rTttj|tj ddd}| |j WddS1sMwYdSdS)aWrite `MetricInfo` as JSON to `metric_info_dir`. Also save the license separately in LICENCE. If `pretty_print` is True, the JSON will be pretty-printed with the indent level of 4. Example: ```py >>> from datasets import load_metric >>> metric = load_metric("accuracy") >>> metric.info.write_to_directory("/path/to/directory/") ``` rrrrNr) rvrrrxr METRIC_INFO_FILENAMErrrrWr{r)r4metric_info_dirrrr>r!r!r"r#s "zMetricInfo.write_to_directoryr7cCshtd||stdttj|tjdd }t |}Wdn1s*wY| |S)aCreate MetricInfo from the JSON file in `metric_info_dir`. Args: metric_info_dir: `str` The directory containing the metadata file. This should be the root directory of a specific dataset version. Example: ```py >>> from datasets import MetricInfo >>> metric_info = MetricInfo.from_directory("/path/to/directory/") ``` zLoading Metric info from zCCalling MetricInfo.from_directory() with undefined metric_info_dir.rrN) rrrrvrrrxr rrrr2)rMrr>metric_info_dictr!r!r"r7s  zMetricInfo.from_directoryrcr8)NcSr9r!r:r<r!r!r"r?Pr@z'MetricInfo.from_dict..crAr!r!rBrEr!r"rGQrHz(MetricInfo.from_dict..r!rI)rMrr!rEr"r2NrNzMetricInfo.from_dictr)r7r)rrrr*rr rrJrrrVrWrlrrrrboolrrrr]rr5rrPrrOr2r!r!r!r"rs*     r)3r*rrJrrrwrsrpathlibrtypingrrrrrfsspec fsspec.corer huggingface_hubr r rr r.rrr_rtasksrrutilsr utils.loggingrutils.py_utilsrrrrrr$ Exceptionr(r,r-rQrrrr!r!r!r"sH       6m