o TZhh@s~dZddlZddlZddlZddlZddlZddlZddlmZddlm Z ddl m Z m Z m Z mZmZddlZddlmZmZddlmZdd lmZmZdd lmZdd lmZmZdd lmZdd l m!Z!ddl"m#Z#m$Z$e!e%Z&eGdddZ'eGdddZ(Gddde)Z*Gddde)Z+eGdddZ,eGdddZ-Gddde e.e-fZ/eGdddZ0dS)aDatasetInfo and MetricInfo record information we know about a dataset and a metric. This includes things that we know about the dataset statically, i.e.: - description - canonical location - does it have validation and tests splits - size - etc. This also includes the things that can and should be computed once we've processed the dataset as well: - number of examples (in each split) - etc. N) dataclass)Path)ClassVarDictListOptionalUnion) DatasetCardDatasetCardData)config)FeaturesValue) SplitDict) TaskTemplatetask_template_from_dict)Version) get_logger)asdict unique_valuesc@&eZdZUdZeed<dZeed<dS)SupervisedKeysDatainputoutputN)__name__ __module__ __qualname__rstr__annotations__rr r D/var/www/html/lang_env/lib/python3.10/site-packages/datasets/info.pyr8 rc@r)DownloadChecksumsEntryDatarkeyvalueN)rrrr$rrr%r r r r!r#>r"r#c@eZdZdZdS)MissingCachedSizesConfigErrorz;The expected cached sizes of the download file are missing.Nrrr__doc__r r r r!r'Dr'c@r&)NonMatchingCachedSizesErrorz/The prepared split doesn't have expected sizes.Nr(r r r r!r+Hr*r+c@sLeZdZUdZeeed<dZeeed<ddZ e deddfdd Z dS) PostProcessedInfoNfeaturesresources_checksumscCs0|jdurt|jtst|j|_dSdSdSN)r- isinstancer from_dictselfr r r! __post_init__QszPostProcessedInfo.__post_init__post_processed_info_dictreturnc4ddt|D|difdd|DS)NcSh|]}|jqSr name.0fr r r! Xz.PostProcessedInfo.from_dict..ci|] \}}|vr||qSr r r<kv field_namesr r! Yz/PostProcessedInfo.from_dict..r  dataclassesfieldsitems)clsr5r rDr!r1V zPostProcessedInfo.from_dict) rrrr-rr rr.dictr4 classmethodr1r r r r!r,Ls r,c@seZdZUdZejedZeed<ejedZ eed<ejedZ eed<ejedZ eed<dZ e eed<dZe eed <dZe eed <dZe eeed <dZe eed <dZe eed <dZe eed<dZe eeefed<dZe eed<dZe eed<dZe e ed<dZ!e e ed<dZ"e e ed<dZ#e e ed<gdZ$e%eeed<ddZ& d8de efddZ'd9dd Z(d!d"Z)e*d#edfd$d%Z+e* d:d&ede ed'dfd(d)Z,e*d*ed'dfd+d,Z-d;d DatasetInfoa Information about a dataset. `DatasetInfo` documents datasets, including its name, version, and features. See the constructor arguments and properties for a full list. Not all fields are known on construction and may be updated later. Attributes: description (`str`): A description of the dataset. citation (`str`): A BibTeX citation of the dataset. homepage (`str`): A URL to the official homepage for the dataset. license (`str`): The dataset's license. It can be the name of the license or a paragraph containing the terms of the license. features ([`Features`], *optional*): The features used to specify the dataset's column types. post_processed (`PostProcessedInfo`, *optional*): Information regarding the resources of a possible post-processing of a dataset. For example, it can contain the information of an index. supervised_keys (`SupervisedKeysData`, *optional*): Specifies the input feature and the label for supervised learning if applicable for the dataset (legacy from TFDS). builder_name (`str`, *optional*): The name of the `GeneratorBasedBuilder` subclass used to create the dataset. Usually matched to the corresponding script name. It is also the snake_case version of the dataset builder class name. config_name (`str`, *optional*): The name of the configuration derived from [`BuilderConfig`]. version (`str` or [`Version`], *optional*): The version of the dataset. splits (`dict`, *optional*): The mapping between split name and metadata. download_checksums (`dict`, *optional*): The mapping between the URL to download the dataset's checksums and corresponding metadata. download_size (`int`, *optional*): The size of the files to download to generate the dataset, in bytes. post_processing_size (`int`, *optional*): Size of the dataset in bytes after post-processing, if any. dataset_size (`int`, *optional*): The combined size in bytes of the Arrow tables for all splits. size_in_bytes (`int`, *optional*): The combined size in bytes of all files associated with the dataset (downloaded files + Arrow files). task_templates (`List[TaskTemplate]`, *optional*): The task templates to prepare the dataset for during training and evaluation. Each template casts the dataset's [`Features`] to standardized column names and types as detailed in `datasets.tasks`. **config_kwargs (additional keyword arguments): Keyword arguments to be passed to the [`BuilderConfig`] and used in the [`DatasetBuilder`]. default_factory descriptioncitationhomepagelicenseNr-post_processedsupervised_keystask_templates builder_name dataset_name config_nameversionsplitsdownload_checksums download_sizepost_processing_size dataset_size size_in_bytes)r\r`rbr-r^_INCLUDED_INFO_IN_YAMLcsjdurtjtstj_jdur$tjts$tj_jdurCtjtsCtjtrsz-DatasetInfo.__post_init__..cSsg|]}|dur|qSr/r rer r r!rgscsg|]}|jqSr )Zalign_with_featuresr-rer2r r!rgs r )r-r0r r1rWr,r]rrr^rZfrom_split_dictrXrtuplelistrYrr)r3Z templatesrfr r2r!r4s@          zDatasetInfo.__post_init__F deprecatedstorage_optionscCs|dkr tdt|j}tj||d\}}}|t|t j d}|j ||dWdn1s4wY|j r_|t|t j d}||WddS1sXwYdSdS)ahWrite `DatasetInfo` and license (if present) as JSON files to `dataset_info_dir`. Args: dataset_info_dir (`str`): Destination directory. pretty_print (`bool`, defaults to `False`): If `True`, the JSON will be pretty-printed with the indent level of 4. fs (`fsspec.spec.AbstractFileSystem`, *optional*): Instance of the remote filesystem used to download the files from. `fs` was deprecated in version 2.9.0 and will be removed in 3.0.0. Please use `storage_options` instead, e.g. `storage_options=fs.storage_options`. storage_options (`dict`, *optional*): Key/value pairs to be passed on to the file-system backend, if any. Example: ```py >>> from datasets import load_dataset >>> ds = load_dataset("rotten_tomatoes", split="validation") >>> ds.info.write_to_directory("/path/to/directory/") ``` rj'fs' was deprecated in favor of 'storage_options' in version 2.9.0 and will be removed in 3.0.0. You can remove this warning by passing 'storage_options=fs.storage_options' instead.rkwb) pretty_printN)warningswarn FutureWarningrkfsspecget_fs_token_pathsopen posixpathjoinr DATASET_INFO_FILENAME _dump_inforVLICENSE_FILENAME _dump_license)r3dataset_info_dirrofsrk_r=r r r!write_to_directorys! "zDatasetInfo.write_to_directorycCs*|tjt||r dnddddS)zQDump info in `file` file-like object open in bytes mode (to support remote files)Nindentutf-8)writejsondumpsrencode)r3fileror r r!rys*zDatasetInfo._dump_infocCs||jddS)zTDump license in `file` file-like object open in bytes mode (to support remote files)rN)rrVr)r3rr r r!r{ szDatasetInfo._dump_license dataset_infosc s(ddDtdkrtfddDrdSdtddD}dtddD}dtd dD}dtd dD}d}d}d}d dD} t| d krwtt| dj| d d}n t| rtt| d}|r|nd}||||||||d S)NcSsg|] }|dur|qSr/)copyr< dset_infor r r!rgz*DatasetInfo.from_merge..rc3s|] }d|kVqdS)rNr rrr r! sz)DatasetInfo.from_merge..z cs|]}|jVqdSr/)rSr<infor r r!rcsrr/)rTrr r r!rrcsrr/)rUrr r r!rrcsrr/)rVrr r r!rrcSsg|] }|jdur|jqSr/)rYrr r r!rgrr )rSrTrUrVr-rXrY)lenallrwrstripriset intersection) rLrrSrTrUrVr-rXrYZall_task_templatesr rr! from_merge s2"   zDatasetInfo.from_merger|r6cCs|dkr tdt|j}tj||d\}}}td||s%td|j t |t j ddd }t|}Wd n1sBwY||S) aCreate [`DatasetInfo`] from the JSON file in `dataset_info_dir`. This function updates all the dynamically generated fields (num_examples, hash, time of creation,...) of the [`DatasetInfo`]. This will overwrite all previous metadata. Args: dataset_info_dir (`str`): The directory containing the metadata file. This should be the root directory of a specific dataset version. fs (`fsspec.spec.AbstractFileSystem`, *optional*): Instance of the remote filesystem used to download the files from. `fs` was deprecated in version 2.9.0 and will be removed in 3.0.0. Please use `storage_options` instead, e.g. `storage_options=fs.storage_options`. storage_options (`dict`, *optional*): Key/value pairs to be passed on to the file-system backend, if any. Example: ```py >>> from datasets import DatasetInfo >>> ds_info = DatasetInfo.from_directory("/path/to/directory/") ``` rjrlrmzLoading Dataset info from zECalling DatasetInfo.from_directory() with undefined dataset_info_dir.rrencodingN)rprqrrrkrsrtloggerr ValueErrorrurvrwr rxrloadr1)rLr|r}rkr~r=dataset_info_dictr r r!from_directory0s%  zDatasetInfo.from_directoryrcr7)NcSr8r r9r;r r r!r>hr?z(DatasetInfo.from_dict..cr@r r rArDr r!rFirGz)DatasetInfo.from_dict..r rH)rLrr rDr!r1frMzDatasetInfo.from_dictTother_dataset_infocs.|j}|jdifdd|jDdS)Ncs(i|]\}}|dus s|t|qSr/rdeepcopyrA ignore_noner r!rFns   z&DatasetInfo.update..r )__dict__updaterK)r3rrZ self_dictr rr!rks   zDatasetInfo.updatecCs |jdidd|jDS)NcSi|] \}}|t|qSr rrAr r r!rFvrz$DatasetInfo.copy..r ) __class__rrKr2r r r!rus zDatasetInfo.copycCsfi}t|}|D](}||jvr0t||}t|dr |||<qt|dr,|||<q|||<q|S)N _to_yaml_list_to_yaml_string)rrdgetattrhasattrrr)r3Z yaml_dictrr$r%r r r! _to_yaml_dictxs    zDatasetInfo._to_yaml_dict yaml_datacs~t|}|ddurt|d|d<|ddur%t|d|d<ddt|D|difdd|DS)Nr-r^cSr8r r9r;r r r!r>r?z.DatasetInfo._from_yaml_dict..cr@r r rArDr r!rFrGz/DatasetInfo._from_yaml_dict..r ) rrgetr Z_from_yaml_listrrIrJrK)rLrr rDr!_from_yaml_dicts  zDatasetInfo._from_yaml_dict)FrjNF)rjN)T)rrP)r6rP)2rrrr)rIfieldrrSrrTrUrVr-rr rWr,rXrrYrrrZr[r\r]rrr^rNr_r`intrarbrcrdrr4rryr{rOrrr1rrrrr r r r!rP\s\ /* 1" 5  rPc@sNeZdZd dddZedddZededdfd d Zdeddfd d ZdS)DatasetInfosDictFr6Nc Cs i}tj|tj}tj|tj}|s||}||tj|rPt |ddd}dd| D}t j |||r=dnddWdn1sKwYtj|r_t |} | j} nd} t} |r|| | duryt dt| d n| } | t|dSdS) NwrrcSsi|] \}}|t|qSr )rr<r\rr r r!rFsz7DatasetInfosDict.write_to_directory..rrz--- z --- )ospathrwr DATASETDICT_INFOS_FILENAMEREPOCARD_FILENAMErrexistsrurKrdumpr rdatar to_dataset_card_datarsaver) r3dataset_infos_dir overwriterototal_dataset_infosZdataset_infos_pathZdataset_readme_pathr=Zdataset_infos_dictZ dataset_carddataset_card_datar r r!rs0      z#DatasetInfosDict.write_to_directorycCstd|tjtj|tjr(t t |tjj }d|vr(| |Stjtj|tj r`ttj|tj dd}|ddt |DWdS1sYwYdS|S)NzLoading Dataset Infos from dataset_inforrcSrr )rPr1)r<r\rr r r!rFs z3DatasetInfosDict.from_directory..)rrrrrrwr rr rrrfrom_dataset_card_datarrurrK)rLrrr=r r r!rs  $zDatasetInfosDict.from_directoryrcCslt|dttfr3t|dtr|dd|dDSt|d}|ddd|_||j|iS|S)NrcSs i|] }|ddt|qS)r\default)rrPr)r<dataset_info_yaml_dictr r r!rFs z;DatasetInfosDict.from_dataset_card_data..r\r)r0rrirNrPrr\)rLrrr r r!rs z'DatasetInfosDict.from_dataset_card_datacCs6|rd|vrt|dtr|ddd|di}nd|vr/t|dtr/dd|dD}ni}i|dd|D}|D]\}}||d<qBt|dkrutt||d<|d dd}|dkrsd|i|d|d<dSdSg|d<t |D]\}}| ddd|i|}|d |qdSdS)Nrr\rcSsi|]}|d|qS)r\r )r<Zconfig_metadatar r r!rFsz9DatasetInfosDict.to_dataset_card_data..cSsi|] \}}||qSr )rrr r r!rFsr ) r0rNrrirKrnextitervaluespopsortedappend)r3rZdataset_metadata_infosrr\Zdset_info_yaml_dictrr r r!rs@     !z%DatasetInfosDict.to_dataset_card_data)FF)r6N)r6r) rrrrrOrr rrr r r r!rs  rc@seZdZUdZeed<eed<eed<ejedZ eed<ejedZ eed<ejedZ eed<eje dZ eeed <eje dZeeed <d Zeed <d Zeeed<d Zeeed<d Zeeed<d Zeeed<ddZdddZedddZededdfddZd S) MetricInfoaInformation about a metric. `MetricInfo` documents a metric, including its name, version, and features. See the constructor arguments and properties for a full list. Note: Not all fields are known on construction and may be updated later. rSrTr-rQinputs_descriptionrUrV codebase_urlsreference_urlsF streamableNformat metric_namer\ experiment_idcCsH|jdur |jD]\}}t|tstd|d|jjq dSdS)NzSWhen using 'numpy' format, all features should be a `datasets.Value` feature. Here z is an instance of )rr-rKr0rrrr)r3r$r%r r r!r4s  zMetricInfo.__post_init__cCsttj|tjddd}tjt|||rdnddWdn1s&wY|j rTttj|tj ddd}| |j WddS1sMwYdSdS)aWrite `MetricInfo` as JSON to `metric_info_dir`. Also save the license separately in LICENCE. If `pretty_print` is True, the JSON will be pretty-printed with the indent level of 4. Example: ```py >>> from datasets import load_metric >>> metric = load_metric("accuracy") >>> metric.info.write_to_directory("/path/to/directory/") ``` rrrrNr) rurrrwr METRIC_INFO_FILENAMErrrrVrzr)r3metric_info_dirror=r r r!r"s "zMetricInfo.write_to_directoryr6cCshtd||stdttj|tjdd }t |}Wdn1s*wY| |S)aCreate MetricInfo from the JSON file in `metric_info_dir`. Args: metric_info_dir: `str` The directory containing the metadata file. This should be the root directory of a specific dataset version. Example: ```py >>> from datasets import MetricInfo >>> metric_info = MetricInfo.from_directory("/path/to/directory/") ``` zLoading Metric info from zCCalling MetricInfo.from_directory() with undefined metric_info_dir.rrN) rrrrurrrwr rrrr1)rLrr=metric_info_dictr r r!r6s  zMetricInfo.from_directoryrcr7)NcSr8r r9r;r r r!r>Or?z'MetricInfo.from_dict..cr@r r rArDr r!rFPrGz(MetricInfo.from_dict..r rH)rLrr rDr!r1MrMzMetricInfo.from_dictr)r6r)rrrr)rrr rIrrrUrVrirrrrboolrrrr\rr4rrOrrNr1r r r r!rs*     r)1r)rrIrrrvrprpathlibrtypingrrrrrrsZhuggingface_hubr r rr r-r rr^rtasksrrutilsrZ utils.loggingrZutils.py_utilsrrrrrr# Exceptionr'r+r,rPrrrr r r r!sF      6m