o +if g@sdZddlZddlZddlZddlZddlmZmZddlm Z m Z m Z m Z m Z mZddlZddlmZddlmZddlmZdd lmZdd lmZdd lmZmZmZmZmZdd l m!Z!dd l"m#Z#ddl$m%Z%ddl&m'Z'e%e(Z)Gddde*Z+e!dd5ddZ,e!dd5ddZ-e!dd6de.de.de efddZ/e!d d6de.de.de efd!d"Z0      #d7de.d$e ee e e.fde ed%e eee.fd&e ee.e'fd'e ee1e.ff d(d)Z2     d8de.d&e ee.e'fde ed%e eee.fd*e e.d$e ee e e.ff d+d,Z3     d8de.d&e ee.e'fde ed%e eee.fd*e e.d$e ee e e.fd-e e.fd.d/Z4       #d9de.d0e e.d$e ee.e e.e e.ee.e e.fffde ed%e eee.fd&e ee.e'fd'e ee1e.fd-efd1d2Z5       #d9de.d0e e.d$e ee.e e.e e.ee.e e.fffde ed%e eee.fd&e ee.e'fd'e ee1e.ffd3d4Z6dS):zList and inspect datasets.N)PathPurePath)DictListMappingOptionalSequenceUnion)config)DownloadConfig) DownloadMode)StreamingDownloadManager) DatasetInfo)dataset_module_factoryget_dataset_builder_classimport_main_classload_dataset_buildermetric_module_factory) deprecated)relative_to_absolute_path) get_logger)Versionc@s eZdZdS)SplitsNotFoundErrorN)__name__ __module__ __qualname__rrI/var/www/html/corbot_env/lib/python3.10/site-packages/datasets/inspect.pyr0srz,Use 'huggingface_hub.list_datasets' instead.TFcCs8tj|d}|sdd|D}|sdd|D}t|S)a]List all the datasets scripts available on the Hugging Face Hub. Args: with_community_datasets (`bool`, *optional*, defaults to `True`): Include the community provided datasets. with_details (`bool`, *optional*, defaults to `False`): Return the full details on the datasets instead of only the short name. Example: ```py >>> from datasets import list_datasets >>> list_datasets() ['acronym_identification', 'ade_corpus_v2', 'adversarial_qa', 'aeslc', 'afrikaans_ner_corpus', 'ag_news', ... ] ``` )fullcSg|] }d|jvr|qS/id.0datasetrrr Oz!list_datasets..cSg|]}|jqSrr#r%rrrr(Q)huggingface_hub list_datasetslist)with_community_datasets with_detailsdatasetsrrrr-4s r-uxUse 'evaluate.list_evaluation_modules' instead, from the new library 🤗 Evaluate: https://huggingface.co/docs/evaluatecCs0t}|s dd|D}|sdd|D}|S)uList all the metrics script available on the Hugging Face Hub. Use `evaluate.list_evaluation_modules` instead, from the new library 🤗 Evaluate: https://huggingface.co/docs/evaluate Args: with_community_metrics (:obj:`bool`, optional, default ``True``): Include the community provided metrics. with_details (:obj:`bool`, optional, default ``False``): Return the full details on the metrics instead of only the short name. Example: ```py >>> from datasets import list_metrics >>> list_metrics() ['accuracy', 'bertscore', 'bleu', 'bleurt', 'cer', 'chrf', ... ] ``` cSr r!r#r&metricrrrr(vr)z list_metrics..cSr*rr#r2rrrr(xr+)r, list_metrics)with_community_metricsr0metricsrrrr4Us r4z?Clone the dataset repository from the Hugging Face Hub instead.path local_pathdownload_configcKs|dur td i|}tj|rtt|j}tj|r'tj ||ddnt j t j |jdj|d||jdtd|d|d t|d dS) a Allow inspection/modification of a dataset script by copying on local drive at local_path. Args: path (`str`): Path to the dataset processing script with the dataset builder. Can be either: - a local path to processing script or the directory containing the script (if the script has the same name as the directory), e.g. `'./dataset/squad'` or `'./dataset/squad/squad.py'`. - a dataset identifier on the Hugging Face Hub (list all available datasets and ids with [`list_datasets`]) e.g. `'squad'`, `'glue'` or `'openai/webtext'`. local_path (`str`): Path to the local folder to copy the dataset script to. download_config ([`DownloadConfig`], *optional*): Specific download configuration parameters. **download_kwargs (additional keyword arguments): Optional arguments for [`DownloadConfig`] which will override the attributes of `download_config` if supplied. NT) dirs_exist_ok)endpointtokenr')repo_id repo_type local_dirforce_downloadz The dataset  can be inspected at z\. You can modify this loading script if it has one and use it with `datasets.load_dataset("")`.r)r osr7isfilestrrparentisdirshutilcopytreer,HfApir HF_ENDPOINTr<snapshot_downloadr@printras_posix)r7r8r9download_kwargsrrrinspect_dataset|s    rPuzUse 'evaluate.inspect_evaluation_module' instead, from the new library 🤗 Evaluate: https://huggingface.co/docs/evaluatec Kst|fd|i|}t|jdd}t|}tj|}t|D]>\}} } tj |tj ||} tj | dddd| D| dd<| D]} t tj || tj | | qFt || q!t|}td |d |d |d t|d dS)u Allow inspection/modification of a metric script by copying it on local drive at local_path. Use `evaluate.inspect_evaluation_module` instead, from the new library 🤗 Evaluate instead: https://huggingface.co/docs/evaluate Args: path (``str``): path to the dataset processing script with the dataset builder. Can be either: - a local path to processing script or the directory containing the script (if the script has the same name as the directory), e.g. ``'./dataset/squad'`` or ``'./dataset/squad/squad.py'`` - a dataset identifier on the Hugging Face Hub (list all available datasets and ids with ``datasets.list_datasets()``) e.g. ``'squad'``, ``'glue'`` or ``'openai/webtext'`` local_path (``str``): path to the local folder to copy the datset script to. download_config (Optional ``datasets.DownloadConfig``): specific download configuration parameters. **download_kwargs (additional keyword arguments): optional attributes for DownloadConfig() which will override the attributes in download_config if supplied. r9F)r'T)exist_okcSsg|] }|ds|qS)).__) startswith)r&dirnamerrrr(r)z"inspect_metric..Nz"The processing scripts for metric rAz. The main class is in zP. You can modify this processing scripts and use it with `datasets.load_metric("rB)rr module_pathinspect getsourcefilerCr7rUwalkjoinrelpathmakedirsrHcopy2copystatrrMrrN) r7r8r9rO metric_module metric_clsmodule_source_pathmodule_source_dirpathdirpathdirnames filenames dst_dirpathfilenamerrrinspect_metrics&  " rhr data_files download_moderevisionr<c  sJ|dkr tdt|td}fdd|DS)a! Get the meta information about a dataset, returned as a dict mapping config name to DatasetInfoDict. Args: path (`str`): path to the dataset processing script with the dataset builder. Can be either: - a local path to processing script or the directory containing the script (if the script has the same name as the directory), e.g. `'./dataset/squad'` or `'./dataset/squad/squad.py'` - a dataset identifier on the Hugging Face Hub (list all available datasets and ids with [`datasets.list_datasets`]) e.g. `'squad'`, `'glue'` or``'openai/webtext'` revision (`Union[str, datasets.Version]`, *optional*): If specified, the dataset module will be loaded from the datasets repository at this version. By default: - it is set to the local version of the lib. - it will also try to load it from the main branch if it's not available at the local version of the lib. Specifying a version that is different from your local version of the lib might cause compatibility issues. download_config ([`DownloadConfig`], *optional*): Specific download configuration parameters. download_mode ([`DownloadMode`] or `str`, defaults to `REUSE_DATASET_IF_EXISTS`): Download/generate mode. data_files (`Union[Dict, List, str]`, *optional*): Defining the data_files of the dataset configuration. token (`str` or `bool`, *optional*): Optional string or boolean to use as Bearer token for remote files on the Datasets Hub. If `True`, or not specified, will get token from `"~/.huggingface"`. use_auth_token (`str` or `bool`, *optional*): Optional string or boolean to use as Bearer token for remote files on the Datasets Hub. If `True`, or not specified, will get token from `"~/.huggingface"`. `use_auth_token` was deprecated in favor of `token` in version 2.14.0 and will be removed in 3.0.0. **config_kwargs (additional keyword arguments): Optional attributes for builder class which will override the attributes if supplied. Example: ```py >>> from datasets import get_dataset_infos >>> get_dataset_infos('rotten_tomatoes') {'default': DatasetInfo(description="Movie Review Dataset. This is a dataset of containing 5,331 positive and 5,331 negative processed sentences from Rotten Tomatoes movie reviews...), ...} ``` r'use_auth_token' was deprecated in favor of 'token' in version 2.14.0 and will be removed in 3.0.0. You can remove this warning by passing 'token=' instead.)r7rkr9rjrir<c s,i|]}|td|dqS))r7 config_namerir9rjrkr<r)get_dataset_config_info)r&rm config_kwargsrir9rjr7rkr<rr s z%get_dataset_infos..)warningswarn FutureWarningget_dataset_config_names) r7rir9rjrkr<use_auth_tokenrp config_namesrrorget_dataset_infoss"7 rxdynamic_modules_pathc KsTt|f|||||d|}t|tj|d}t|jp)|j d|j p'dgS)aGet the list of available config names for a particular dataset. Args: path (`str`): path to the dataset processing script with the dataset builder. Can be either: - a local path to processing script or the directory containing the script (if the script has the same name as the directory), e.g. `'./dataset/squad'` or `'./dataset/squad/squad.py'` - a dataset identifier on the Hugging Face Hub (list all available datasets and ids with [`datasets.list_datasets`]) e.g. `'squad'`, `'glue'` or `'openai/webtext'` revision (`Union[str, datasets.Version]`, *optional*): If specified, the dataset module will be loaded from the datasets repository at this version. By default: - it is set to the local version of the lib. - it will also try to load it from the main branch if it's not available at the local version of the lib. Specifying a version that is different from your local version of the lib might cause compatibility issues. download_config ([`DownloadConfig`], *optional*): Specific download configuration parameters. download_mode ([`DownloadMode`] or `str`, defaults to `REUSE_DATASET_IF_EXISTS`): Download/generate mode. dynamic_modules_path (`str`, defaults to `~/.cache/huggingface/modules/datasets_modules`): Optional path to the directory in which the dynamic modules are saved. It must have been initialized with `init_dynamic_modules`. By default the datasets and metrics are stored inside the `datasets_modules` module. data_files (`Union[Dict, List, str]`, *optional*): Defining the data_files of the dataset configuration. **download_kwargs (additional keyword arguments): Optional attributes for [`DownloadConfig`] which will override the attributes in `download_config` if supplied, for example `token`. Example: ```py >>> from datasets import get_dataset_config_names >>> get_dataset_config_names("glue") ['cola', 'sst2', 'mrpc', 'qqp', 'stsb', 'mnli', 'mnli_mismatched', 'mnli_matched', 'qnli', 'rte', 'wnli', 'ax'] ``` rkr9rjryri dataset_namermdefault) rrrCr7basenamer.builder_configskeysbuilder_kwargsgetDEFAULT_CONFIG_NAME) r7rkr9rjryrirOdataset_module builder_clsrrrru#s8 rureturnc Ksjt|f|||||d|}t|tj|d}t|j} | r.t| dkr+| dnd} nd} |j p4| S)a Get the default config name for a particular dataset. Can return None only if the dataset has multiple configurations and no default configuration. Args: path (`str`): path to the dataset processing script with the dataset builder. Can be either: - a local path to processing script or the directory containing the script (if the script has the same name as the directory), e.g. `'./dataset/squad'` or `'./dataset/squad/squad.py'` - a dataset identifier on the Hugging Face Hub (list all available datasets and ids with [`datasets.list_datasets`]) e.g. `'squad'`, `'glue'` or `'openai/webtext'` revision (`Union[str, datasets.Version]`, *optional*): If specified, the dataset module will be loaded from the datasets repository at this version. By default: - it is set to the local version of the lib. - it will also try to load it from the main branch if it's not available at the local version of the lib. Specifying a version that is different from your local version of the lib might cause compatibility issues. download_config ([`DownloadConfig`], *optional*): Specific download configuration parameters. download_mode ([`DownloadMode`] or `str`, defaults to `REUSE_DATASET_IF_EXISTS`): Download/generate mode. dynamic_modules_path (`str`, defaults to `~/.cache/huggingface/modules/datasets_modules`): Optional path to the directory in which the dynamic modules are saved. It must have been initialized with `init_dynamic_modules`. By default the datasets and metrics are stored inside the `datasets_modules` module. data_files (`Union[Dict, List, str]`, *optional*): Defining the data_files of the dataset configuration. **download_kwargs (additional keyword arguments): Optional attributes for [`DownloadConfig`] which will override the attributes in `download_config` if supplied, for example `token`. Returns: Optional[str]: the default config name if there is one Example: ```py >>> from datasets import get_dataset_default_config_name >>> get_dataset_default_config_name("openbookqa") 'main' ``` rzr{r rNr}) rrrCr7r~r.rrlenr) r7rkr9rjryrirOrrrdefault_config_namerrrget_dataset_default_config_namejs"1  rrmc s|dkr tdt|}tf||||||d|} | j} | jdurd|r)|nt}|dur3||_| t | j |dzfdd| t | j |dD| _W| St yc} ztd| d} ~ ww| S) aGet the meta information (DatasetInfo) about a dataset for a particular config Args: path (``str``): path to the dataset processing script with the dataset builder. Can be either: - a local path to processing script or the directory containing the script (if the script has the same name as the directory), e.g. ``'./dataset/squad'`` or ``'./dataset/squad/squad.py'`` - a dataset identifier on the Hugging Face Hub (list all available datasets and ids with ``datasets.list_datasets()``) e.g. ``'squad'``, ``'glue'`` or ``'openai/webtext'`` config_name (:obj:`str`, optional): Defining the name of the dataset configuration. data_files (:obj:`str` or :obj:`Sequence` or :obj:`Mapping`, optional): Path(s) to source data file(s). download_config (:class:`~download.DownloadConfig`, optional): Specific download configuration parameters. download_mode (:class:`DownloadMode` or :obj:`str`, default ``REUSE_DATASET_IF_EXISTS``): Download/generate mode. revision (:class:`~utils.Version` or :obj:`str`, optional): Version of the dataset script to load. As datasets have their own git repository on the Datasets Hub, the default version "main" corresponds to their "main" branch. You can specify a different version than the default "main" by using a commit SHA or a git tag of the dataset repository. token (``str`` or :obj:`bool`, optional): Optional string or boolean to use as Bearer token for remote files on the Datasets Hub. If True, or not specified, will get token from `"~/.huggingface"`. use_auth_token (``str`` or :obj:`bool`, optional): Optional string or boolean to use as Bearer token for remote files on the Datasets Hub. If True, or not specified, will get token from `"~/.huggingface"`. `use_auth_token` was deprecated in favor of `token` in version 2.14.0 and will be removed in 3.0.0. **config_kwargs (additional keyword arguments): optional attributes for builder class which will override the attributes if supplied. rrl)namerir9rjrkr<N) base_pathr9csi|] }|j|jdqS))rr|)r)r&split_generatorr7rrrqsz+get_dataset_config_info..z `use_auth_token` was deprecated in favor of `token` in version 2.14.0 and will be removed in 3.0.0. **config_kwargs (additional keyword arguments): Optional attributes for builder class which will override the attributes if supplied. Example: ```py >>> from datasets import get_dataset_split_names >>> get_dataset_split_names('rotten_tomatoes') ['train', 'validation', 'test'] ``` rrl)rmrir9rjrkr<)rrrsrtrnr.rr) r7rmrir9rjrkr<rvrprrrrget_dataset_split_namess&8 r)TF)N)NNNNNr)NNNNN)NNNNNNr)7__doc__rWrCrHrrpathlibrrtypingrrrrrr r,r download.download_configr download.download_managerr #download.streaming_download_managerrrrloadrrrrrutils.deprecation_utilsrutils.file_utilsr utils.loggingr utils.versionrrlogger ValueErrorrr-r4rErPrhboolrxrurrnrrrrrs             $$+ X I E( Q(