o TZhi| @sddlZddlZddlmZddlmZddlmZmZddl m Z m Z m Z m Z mZmZmZddlZddlmZddlmZddlmZdd lmZdd lmZd d lmZd d lmZd dlm Z m!Z!m"Z"d dl#m$Z$d dl%m&Z&d dl'm(Z(d dl'm)Z*d dl+m,Z,m-Z-d dl.m/Z/m0Z0e1e&j2Z3e(4e5Z6Gddde1Z7Gddde8Z9dZ:e&j2ddge&j;gde&jej?e@dkrd d!gZAnej?e@d"krd d#gZAngd$ZAe&j2e&j;e&jZP  dVd-e1d?e1d@e e e1dAe ed.e e1f dBdCZQdWd?e1dAe ed.e e1e e1ffdDdEZR dWd?e1dAe ed.e e1fdFdGZS dWdHe1dAe ed.ee1fdIdJZT K dXdLe e1dAe ed.ee1fdMdNZUGdOd2d2e e1ZVGdPdQdQe e1eVfZWGdRdSdSe e1ZXGdTdUdUe e1eXfZYdS)YN)partial) has_magic)PathPurePath)CallableDictListOptionalSetTupleUnion)get_fs_token_paths)HTTPFileSystem) HfFileSystem)version) thread_map)config)DownloadConfig)!_prepare_path_and_storage_options xbasenamexjoin) _split_re)Split)logging)tqdm) is_local_pathis_relative_path)glob_pattern_to_regexstring_to_dictc@ eZdZdS)UrlN__name__ __module__ __qualname__r&r&J/var/www/html/lang_env/lib/python3.10/site-packages/datasets/data_files.pyr! r!c@r )EmptyDatasetErrorNr"r&r&r&r'r)$r(r)zFdata/{split}-[0-9][0-9][0-9][0-9][0-9]-of-[0-9][0-9][0-9][0-9][0-9]*.*trainZtraining)Z validationZvaliddevval)testtestingevalZ evaluationz-._ 0-9z2023.9.0z{keyword}[{sep}/]**z**[{sep}/]{keyword}[{sep}/]**z 2023.12.0z**/*[{sep}/]{keyword}[{sep}/]**)z**/{keyword}[{sep}]*z**/{keyword}/**z**/*[{sep}]{keyword}[{sep}]*z**/*[{sep}]{keyword}[{sep}]*/**z**/{keyword}[{sep}]*/**z**/*[{sep}]{keyword}/**cCs i|] }|ddt|DqS)cSs$g|]}tD] }|j|tdqqS))keywordsep)#KEYWORDS_IN_PATH_NAME_BASE_PATTERNSformatNON_WORDS_CHARS).0r0patternr&r&r' @s z.)SPLIT_KEYWORDSr5splitr&r&r' ?s r;z**)z metadata.csv**/metadata.csvzmetadata.jsonl**/metadata.jsonlr<r=z*[])z README.mdz config.jsonzdataset_info.jsonzdataset_infos.jsonzdummy_data.zipzdataset_dict.jsonr6returncstfddtDS)Nc3s|]}|vVqdSNr&)r5Zwilcard_characterr6r&r' iz%contains_wildcards..)anyWILDCARD_CHARACTERSr@r&r@r'contains_wildcardshsrEpatterns DataFilesListcCst|trdd|DSt|trt|giSt|trntdd|Drj|D]"}t|trCt|dkrCd|vrCt|dttfsJt d|q(d d |D}tt |t|krct d |d d|DSt|iSt t|S) a/ Take the data_files patterns from the user, and format them into a dictionary. Each key is the name of the split, and each value is a list of data files patterns (paths or urls). The default split is "train". Returns: patterns: dictionary of split_name -> list of patterns cSs*i|]\}}t|t|tr|n|gqSr&str isinstancelist)r5keyvaluer&r&r'r;vs*z%sanitize_patterns..css|]}t|tVqdSr?)rJdictr5r6r&r&r'rAzsz$sanitize_patterns..r:pathz]Expected each split to have a 'path' key which can be a string or a list of strings, but got cSsg|]}|dqSr:r&rOr&r&r'r7z%sanitize_patterns..z*Some splits are duplicated in data_files: cSs6i|]}t|dt|dtr|dn|dgqS)r:rQrHrOr&r&r'r;s() rJrNitemsrISANITIZED_DEFAULT_SPLITrKrClenget ValueErrorsetsanitize_patterns)rFr6splitsr&r&r'rZls2      rZmatched_rel_pathcCs<ddt|jjD}ddt|jjD}t|t|kS)u When a path matches a pattern, we additionnally check if it's inside a special directory we ignore by default (if it starts with a double underscore). Users can still explicitly request a filepath inside such a directory if "__pycache__" is mentioned explicitly in the requested pattern. Some examples: base directory: ./ └── __pycache__ └── b.txt >>> _is_inside_unrequested_special_dir("__pycache__/b.txt", "**") True >>> _is_inside_unrequested_special_dir("__pycache__/b.txt", "*/b.txt") True >>> _is_inside_unrequested_special_dir("__pycache__/b.txt", "__pycache__/*") False >>> _is_inside_unrequested_special_dir("__pycache__/b.txt", "__*/*") False cSg|] }|dr|qS__ startswithr5partr&r&r'r7z6_is_inside_unrequested_special_dir..cSr]r^r`rbr&r&r'r7rd)rparentpartsrV)r\r6Zdata_dirs_to_ignore_in_pathZdata_dirs_to_ignore_in_patternr&r&r'"_is_inside_unrequested_special_dirsrgcCs8ddt|jD}ddt|jD}t|t|kS)u: When a path matches a pattern, we additionnally check if it's a hidden file or if it's inside a hidden directory we ignore by default, i.e. if the file name or a parent directory name starts with a dot. Users can still explicitly request a filepath that is hidden or is inside a hidden directory if the hidden part is mentioned explicitly in the requested pattern. Some examples: base directory: ./ └── .hidden_file.txt >>> _is_unrequested_hidden_file_or_is_inside_unrequested_hidden_dir(".hidden_file.txt", "**") True >>> _is_unrequested_hidden_file_or_is_inside_unrequested_hidden_dir(".hidden_file.txt", ".*") False base directory: ./ └── .hidden_dir └── a.txt >>> _is_unrequested_hidden_file_or_is_inside_unrequested_hidden_dir(".hidden_dir/a.txt", "**") True >>> _is_unrequested_hidden_file_or_is_inside_unrequested_hidden_dir(".hidden_dir/a.txt", ".*/*") False >>> _is_unrequested_hidden_file_or_is_inside_unrequested_hidden_dir(".hidden_dir/a.txt", ".hidden_dir/*") False base directory: ./ └── .hidden_dir └── .hidden_file.txt >>> _is_unrequested_hidden_file_or_is_inside_unrequested_hidden_dir(".hidden_dir/.hidden_file.txt", "**") True >>> _is_unrequested_hidden_file_or_is_inside_unrequested_hidden_dir(".hidden_dir/.hidden_file.txt", ".*/*") True >>> _is_unrequested_hidden_file_or_is_inside_unrequested_hidden_dir(".hidden_dir/.hidden_file.txt", ".*/.*") False >>> _is_unrequested_hidden_file_or_is_inside_unrequested_hidden_dir(".hidden_dir/.hidden_file.txt", ".hidden_dir/*") True >>> _is_unrequested_hidden_file_or_is_inside_unrequested_hidden_dir(".hidden_dir/.hidden_file.txt", ".hidden_dir/.*") False cS(g|]}|drt|dhks|qS.rarYrbr&r&r'r7 zS_is_unrequested_hidden_file_or_is_inside_unrequested_hidden_dir..cSrhrirkrbr&r&r'r7rl)rrfrV)r\r6Zhidden_directories_in_pathZhidden_directories_in_patternr&r&r'?_is_unrequested_hidden_file_or_is_inside_unrequested_hidden_dirs5rmpattern_resolverc sFtD]Wdd}z||}Wn tyYqwt|dkrYfdd|DtddDr=tdtd d fd d tDtt t}fd d|DSqt D]<g} D]&\}}|D]}z||}Wn ty{Yqjwt|dkr| |nqjqd|rfdd|DSq\td|d|)a+ Get the default pattern from a directory or repository by testing all the supported patterns. The first patterns to return a non-empty list of data files is returned. In order, it first tests if SPLIT_PATTERN_SHARDED works, otherwise it tests the patterns in ALL_DEFAULT_PATTERNS. z{split}*rcs&h|]}tt|ttdqSrR)rrr)r5p split_patternr&r' sz+_get_data_files_patterns..css|] }tt| VqdSr?)rematchrr9r&r&r'rAsz+_get_data_files_patterns..zSplit name should match 'z '' but got 'z'.csg|] }|vrt|qSr&)rIr9)r[r&r'r7z,_get_data_files_patterns..csi|] }|j|dgqS)rR)r3r9rqr&r'r; rvz,_get_data_files_patterns..csi|]}||qSr&r&r9) patterns_dictr&r'r;sCouldn't resolve pattern  with resolver ) ALL_SPLIT_PATTERNSreplaceFileNotFoundErrorrVrCrXrDEFAULT_SPLITSsortedrYALL_DEFAULT_PATTERNSrTappend)rnr6 data_filesZ sorted_splitsZnon_empty_splitsr:rFr&)rwrrr[r'_get_data_files_patternssH           rc Cs^g}tD]}z||}t|dkr||Wqty Yqw|r%|Std|d|)zM Get the supported metadata patterns from a directory or repository. rrxry)METADATA_PATTERNSrVrr|)rnZnon_empty_patternsr6Zmetadata_filesr&r&r'_get_metadata_files_patternss   r base_pathallowed_extensionsdownload_configc st|r t||}nt|rtj|dtj}nd}t||d\}}t||d\}}}| dd ddp;|j | dd ddt t t |ht|jtrZ|jn|jd}|dkrg|dndi}|d kr{tjtd kr{d |d <fd d|j|fddi|D} durfdd| D} t| t| krtt | t | } td|d| n| } | sd|d} dur| dt7} t| | S)a{ Resolve the paths and URLs of the data files from the pattern passed by the user. You can use patterns to resolve multiple local files. Here are a few examples: - *.csv to match all the CSV files at the first level - **.csv to match all the CSV files at any level - data/* to match all the files inside "data" - data/** to match all the files inside "data" and its subdirectories The patterns are resolved using the fsspec glob. In fsspec>=2023.12.0 this is equivalent to Python's glob.glob, Path.glob, Path.match and fnmatch where ** is unsupported with a prefix/suffix other than a forward slash /. More generally: - '*' matches any character except a forward-slash (to match just the file or directory name) - '**' matches any character including a forward-slash / Hidden files and directories (i.e. whose names start with a dot) are ignored, unless they are explicitly requested. The same applies to special directories that start with a double underscore like "__pycache__". You can still include one if the pattern explicilty mentions it: - to include a hidden file: "*/.hidden.txt" or "*/.*" - to include a hidden directory: ".hidden/*" or ".*/*" - to include a special directory: "__special__/*" or "__*/*" Example:: >>> from datasets.data_files import resolve_pattern >>> base_path = "." >>> resolve_pattern("docs/**/*.py", base_path) [/Users/mariosasko/Desktop/projects/datasets/docs/source/_config.py'] Args: pattern (str): Unix pattern or paths or URLs of the data files to resolve. The paths can be absolute or relative to base_path. Remote filesystems using fsspec are supported, e.g. with the hf:// protocol. base_path (str): Base path to use when resolving relative paths. allowed_extensions (Optional[list], optional): White-list of file extensions to use. Defaults to None (all extensions). For example: allowed_extensions=[".csv", ".json", ".txt", ".parquet"] Returns: List[str]: List of paths or URLs to the local or remote files that match the patterns. rrstorage_optionsz::z://fileZhfz0.20.0FZ expand_infocszg|]9\}}|ddkr;t|vrttj|tjsttj|tjs|r7|n|qS)typer)rrgosrQrelpathrmra)r5filepathinfo)files_to_ignore fs_base_path fs_patternprotocol_prefixr&r'r7ls  z#resolve_pattern..detailTNcs8g|]}tfddt|dddDr|qS)c3s|] }d|vVqdS)rjNr&)r5suffixrr&r'rA|sz-resolve_pattern...rjrN)rCrr:)r5rrr&r'r7ys &z Some files matched the pattern 'z-' but don't have valid data file extensions: zUnable to find ''z with any supported extension )rrrrrQ splitdriver1rr r:Z root_markerrYFILES_TO_IGNORErrJprotocolrIrZHF_HUB_VERSIONrparseglobrTrVrKloggerrr|) r6rrrrfs_rZ glob_kwargsZ matched_pathsoutZinvalid_matched_files error_msgr&)rrrrrr'resolve_pattern,sF/   rcCs:tt||d}zt|WStytd|ddw)uh Get the default pattern from a directory testing all the supported patterns. The first patterns to return a non-empty list of data files is returned. Some examples of supported patterns: Input: my_dataset_repository/ ├── README.md └── dataset.csv Output: {"train": ["**"]} Input: my_dataset_repository/ ├── README.md ├── train.csv └── test.csv my_dataset_repository/ ├── README.md └── data/ ├── train.csv └── test.csv my_dataset_repository/ ├── README.md ├── train_0.csv ├── train_1.csv ├── train_2.csv ├── train_3.csv ├── test_0.csv └── test_1.csv Output: {'train': ['train[-._ 0-9/]**', '**/*[-._ 0-9/]train[-._ 0-9/]**', 'training[-._ 0-9/]**', '**/*[-._ 0-9/]training[-._ 0-9/]**'], 'test': ['test[-._ 0-9/]**', '**/*[-._ 0-9/]test[-._ 0-9/]**', 'testing[-._ 0-9/]**', '**/*[-._ 0-9/]testing[-._ 0-9/]**', ...]} Input: my_dataset_repository/ ├── README.md └── data/ ├── train/ │ ├── shard_0.csv │ ├── shard_1.csv │ ├── shard_2.csv │ └── shard_3.csv └── test/ ├── shard_0.csv └── shard_1.csv Output: {'train': ['train[-._ 0-9/]**', '**/*[-._ 0-9/]train[-._ 0-9/]**', 'training[-._ 0-9/]**', '**/*[-._ 0-9/]training[-._ 0-9/]**'], 'test': ['test[-._ 0-9/]**', '**/*[-._ 0-9/]test[-._ 0-9/]**', 'testing[-._ 0-9/]**', '**/*[-._ 0-9/]testing[-._ 0-9/]**', ...]} Input: my_dataset_repository/ ├── README.md └── data/ ├── train-00000-of-00003.csv ├── train-00001-of-00003.csv ├── train-00002-of-00003.csv ├── test-00000-of-00001.csv ├── random-00000-of-00003.csv ├── random-00001-of-00003.csv └── random-00002-of-00003.csv Output: {'train': ['data/train-[0-9][0-9][0-9][0-9][0-9]-of-[0-9][0-9][0-9][0-9][0-9]*.*'], 'test': ['data/test-[0-9][0-9][0-9][0-9][0-9]-of-[0-9][0-9][0-9][0-9][0-9]*.*'], 'random': ['data/random-[0-9][0-9][0-9][0-9][0-9]-of-[0-9][0-9][0-9][0-9][0-9]*.*']} In order, it first tests if SPLIT_PATTERN_SHARDED works, otherwise it tests the patterns in ALL_DEFAULT_PATTERNS. rrThe directory at z doesn't contain any data filesN)rrrr|r)rrresolverr&r&r'get_data_patternss T  rcCs:tt||d}zt|WStytd|ddw)zE Get the supported metadata patterns from a local directory. rrz" doesn't contain any metadata fileN)rrrr|rr&r&r'get_metadata_patternss   r data_filec Cst||d\}}t||d\}}}t|tr!||}|j|jfSt|trQ|t j rQtt j |j d}d|t t j dd ddd}||}|j|jfS||}dD]}||vrgt||fSqXd S) Nrr)Zendpointtokenzhf://rz /resolve/@)ETagetagmtimer&)rr rJrZ resolve_pathZrepo_idrevisionrrarZ HF_ENDPOINTrrVr{rrI) rrrrr resolved_pathZhffsrrLr&r&r'_get_single_origin_metadatas    $   r@rcCs(ttt|d||tdt|dkpddS)NrzResolving data files) max_workersZ tqdm_classZdescdisable)rrrhf_tqdmrV)rrrr&r&r'_get_origin_metadata s rcseZdZdZdeedeeeffdd ZddZe   dd eed e j j d e ed e eed e eddf ddZe   dd eed e ed e eed e eddf ddZe   dd eed e ed e eed e eddf ddZdeeddfddZZS)rGa List of data files (absolute local paths or URLs). It has two construction methods given the user's data files patterns : - ``from_hf_repo``: resolve patterns inside a dataset repository - ``from_local_or_remote``: resolve patterns from a local path Moreover DataFilesList has an additional attribute ``origin_metadata``. It can store: - the last modified time of local files - ETag of remote files - commit sha of a dataset repository Thanks to this additional attribute, it is possible to hash the list and get a different hash if and only if at least one file changed. This is useful for caching Dataset objects that are obtained from a list of data files. rorigin_metadatact|||_dSr?)super__init__r)selfrr __class__r&r'r.s  zDataFilesList.__init__cCtg|||j|jSr?)rGrrotherr&r&r'__add__2zDataFilesList.__add__NrF dataset_inforrrr>cCs6d|jd|jd|p dd}|j||||dS)Nzhf://datasets/r/rrrr)idsharstrip from_patterns)clsrFrrrrr&r&r' from_hf_repo5s$ zDataFilesList.from_hf_repocCs,|dur|nt}|j||||dSNr)rresolveas_posixr)rrFrrrr&r&r'from_local_or_remoteCsz"DataFilesList.from_local_or_remotec Cst|dur|nt}g}|D]}z |t||||dWqty.t|s,Yqwt||d}|||SNrr)rrrextendrr|rr)rrFrrrrr6rr&r&r'rPs&   zDataFilesList.from_patterns extensionscsBddd|Dtddtfdd|D|jdS) N|css|]}d|VqdS)\Nr&)r5extr&r&r'rAkrBz2DataFilesList.filter_extensions..z.*(z )(\..+)?$csg|] }|r|qSr&)ru)r5rr@r&r'r7nrdz3DataFilesList.filter_extensions..)r)joinrtcompilerGrrrr&r@r'filter_extensionsjs zDataFilesList.filter_extensionsNNN)r#r$r%__doc__rrIr rr classmethodhuggingface_hubhf_api DatasetInfor rrrrr __classcell__r&r&rr'rGsh"        c@seZdZdZe   ddeeeeee ffde ede eede e ddf dd Z e   ddeeeeee ffd e jjde ede eede e ddf d d Ze   ddeeeeee ffde ede eede e ddf d dZdeeddfddZdS) DataFilesDicta Dict of split_name -> list of data files (absolute local paths or URLs). It has two construction methods given the user's data files patterns : - ``from_hf_repo``: resolve patterns inside a dataset repository - ``from_local_or_remote``: resolve patterns from a local path Moreover each list is a DataFilesList. It is possible to hash the dictionary and get a different hash if and only if at least one file changed. For more info, see ``DataFilesList``. This is useful for caching Dataset objects that are obtained from a list of data files. Changing the order of the keys of this dictionary also doesn't change its hash. NrFrrrr>cC@|}|D]\}}t|tstj||||dn|||<q|Sr)rTrJrGrrrFrrrrrLpatterns_for_keyr&r&r'r z"DataFilesDict.from_local_or_remoterc CsB|}|D]\}}t|tstj|||||dn|||<q|S)N)rrrr)rTrJrGr) rrFrrrrrrLrr&r&r'rs   zDataFilesDict.from_hf_repocCrr)rTrJrGrrr&r&r'rrzDataFilesDict.from_patternsrcC.t|}|D] \}}||||<q |Sr?rrTr)rrrrLZdata_files_listr&r&r'r zDataFilesDict.filter_extensionsr)r#r$r%rrrrIr rrGr rrrrrrrrr&r&r&r'rssd      rcseZdZdZdeedeeeeffdd ZddZe ddeedeeed d fd d Z dd edee d dfddZ deed dfddZ ZS)DataFilesPatternsListz List of data files patterns (absolute local paths or URLs). For each pattern there should also be a list of allowed extensions to keep, or a None ot keep all the files for the pattern. rFrcrr?)rrr)rrFrrr&r'rs  zDataFilesPatternsList.__init__cCrr?)rGrrr&r&r'rrzDataFilesPatternsList.__add__Nr>DataFilesPatternsDictcCs|||gt|Sr?)rV)rrFrr&r&r'rsz#DataFilesPatternsList.from_patternsrrrGc Cs|dur|nt}g}t||jD]\}}z |t||||dWqty4t|s2Yqwt ||d}t ||Sr) rrrziprrrr|rrrG)rrrrr6rrr&r&r'rs&   zDataFilesPatternsList.resolvercst|fdd|jDS)Ncsg|]}|qSr&r&)r5rrr&r'r7rSz;DataFilesPatternsList.filter_extensions..)rrrr&rr'rsz'DataFilesPatternsList.filter_extensionsr?)r#r$r%rrrIr rrrrrrrrr&r&rr'rs4   rc@sveZdZdZe ddeeeefdeeeddfddZ dded ee dd fd d Z d eeddfddZ dS)rz[ Dict of split_name -> list of data files patterns (absolute local paths or URLs). NrFrr>cCs<|}|D]\}}t|tstj||dn|||<q|S)Nr)rTrJrr)rrFrrrLrr&r&r'rsz#DataFilesPatternsDict.from_patternsrrrcCs,t}|D] \}}|||||<q|Sr?)rrTr)rrrrrLdata_files_patterns_listr&r&r'rszDataFilesPatternsDict.resolvercCrr?r)rrrrLrr&r&r'r"rz'DataFilesPatternsDict.filter_extensionsr?) r#r$r%rrrrIrr rrrrr&r&r&r'rs(    r)NNr?)rN)Zrrt functoolsrrrpathlibrrtypingrrrr r r r rZfsspecr Zfsspec.implementations.httprr packagingrZtqdm.contrib.concurrentrrrdownloadrZ#download.streaming_download_managerrrrZnamingrr[rutilsrrrZutils.file_utilsrrZutils.py_utilsrrrIZTRAINrUZ get_loggerr#rr!r|r)ZSPLIT_PATTERN_SHARDEDZ VALIDATIONTESTr8r4ZFSSPEC_VERSIONrr2r}Z#DEFAULT_PATTERNS_SPLIT_IN_PATH_NAMEZDEFAULT_PATTERNS_ALLrzrrrDrboolrErZrgrmrrrrrrrrGrrrr&r&r&r's  $                      0&!,>$*  (a]   W[5