o +if} @s(ddlZddlZddlmZddlmZddlmZmZddl m Z m Z m Z m Z mZmZmZddlZddlmZddlmZddlmZdd lmZdd lmZd d lmZd d lmZd dlm Z d dl!m"Z"d dl#m$Z$d dl#m%Z&d dl'm(Z(m)Z)m*Z*m+Z+m,Z,d dl-m.Z.m/Z/e0e"j1Z2e$3e4Z5Gddde0Z6Gddde7Z8dZ9e"j1ddge"j:gde"j;gdiZe?dkrdd gZ@gd!ZAnej>e?d"krd#d gZ@gd$ZAnd%d&gZ@gd'ZAe"j1e"j:e"j;gZBd(d)eBDZCd*d)eBDZDe"j1d+giZEe9gZFeDeCeEgZGej>e?dkrgd,ZHnd-d.gZHd/ZIgd0ZJd1e0d2eKfd3d4ZLd5ee e e0fd2e e0ee e0d6fffd7d8ZMd9e0d1e0d2eKfd:d;ZNd9e0d1e0d2eKfde e0ge e0fd2e e0e e0ffd?d@ZPd>e e0ge e0fd2e e0fdAdBZQ  dZd1e0dCe0dDe e e0dEe ed2e e0f dFdGZRd[dCe0dEe ed2e e0e e0ffdHdIZS d[dCe0dEe ed2e e0fdJdKZT d[dLe0dEe ed2ee0fdMdNZU  dZdOe e0dEe edPe eVd2ee0fdQdRZWGdSd6d6e e0ZXGdTdUdUe e0eXfZYGdVdWdWe e0ZZGdXdYdYe e0eZfZ[dS)\N)partial) has_magic)PathPurePath)CallableDictListOptionalSetTupleUnion) url_to_fs)HTTPFileSystem) HfFileSystem)version) thread_map)config)DownloadConfig) _split_re)Split)logging)tqdm)!_prepare_path_and_storage_options is_local_pathis_relative_path xbasenamexjoin)glob_pattern_to_regexstring_to_dictc@ eZdZdS)UrlN__name__ __module__ __qualname__r&r&L/var/www/html/corbot_env/lib/python3.10/site-packages/datasets/data_files.pyr!r!c@r )EmptyDatasetErrorNr"r&r&r&r'r)#r(r)zFdata/{split}-[0-9][0-9][0-9][0-9][0-9]-of-[0-9][0-9][0-9][0-9][0-9]*.*traintraining) validationvaliddevval)testtestingeval evaluationz-._ 0-9z2023.9.0z**[{sep}/]{keyword}[{sep}]*z{keyword}[{sep}]*)z {keyword}/**z{keyword}[{sep}]*/**z**[{sep}/]{keyword}/**z**[{sep}/]{keyword}[{sep}]*/**z 2023.12.0z**/*[{sep}/]{keyword}[{sep}]*)z{keyword}/**/*z{keyword}[{sep}]*/**/*z**/*[{sep}/]{keyword}/**/*z"**/*[{sep}/]{keyword}[{sep}]*/**/*z**/{keyword}[{sep}]*z**/*[{sep}]{keyword}[{sep}]*)z**/{keyword}/**z**/{keyword}[{sep}]*/**z**/*[{sep}]{keyword}/**z**/*[{sep}]{keyword}[{sep}]*/**cC i|] }|ddt|DqS)cS$g|]}tD] }|j|tdqqS)keywordsep)"KEYWORDS_IN_FILENAME_BASE_PATTERNSformatNON_WORDS_CHARS.0r7patternr&r&r' J .SPLIT_KEYWORDSr=splitr&r&r' I rFcCr4)cSr5r6)"KEYWORDS_IN_DIR_NAME_BASE_PATTERNSr:r;r<r&r&r'r?Rr@rArBrDr&r&r'rFQrGz**)z metadata.csv**/metadata.csvzmetadata.jsonl**/metadata.jsonlrIrJz*[])z README.mdz config.jsonzdataset_info.jsonzdataset_infos.jsonzdummy_data.zipzdataset_dict.jsonr>returncstfddtDS)Nc3s|]}|vVqdSNr&)r=wilcard_characterr>r&r' }z%contains_wildcards..)anyWILDCARD_CHARACTERSrNr&rNr'contains_wildcards|srSpatterns DataFilesListcCst|trdd|DSt|trt|giSt|trntdd|Drj|D]"}t|trCt|dkrCd|vrCt|dttfsJt d|q(d d |D}tt |t|krct d |d d|DSt|iSt t|S) a/ Take the data_files patterns from the user, and format them into a dictionary. Each key is the name of the split, and each value is a list of data files patterns (paths or urls). The default split is "train". Returns: patterns: dictionary of split_name -> list of patterns cSs*i|]\}}t|t|tr|n|gqSr&str isinstancelist)r=keyvaluer&r&r'rFs*z%sanitize_patterns..css|]}t|tVqdSrL)rXdictr=r>r&r&r'rOsz$sanitize_patterns..rEpathz]Expected each split to have a 'path' key which can be a string or a list of strings, but got cSsg|]}|dqSrEr&r]r&r&r'r?z%sanitize_patterns..z*Some splits are duplicated in data_files: cSs6i|]}t|dt|dtr|dn|dgqS)rEr_rVr]r&r&r'rFs() rXr\itemsrWSANITIZED_DEFAULT_SPLITrYrQlenget ValueErrorsetsanitize_patterns)rTr>splitsr&r&r'rhs2      rhmatched_rel_pathcCs<ddt|jjD}ddt|jjD}t|t|kS)u When a path matches a pattern, we additionnally check if it's inside a special directory we ignore by default (if it starts with a double underscore). Users can still explicitly request a filepath inside such a directory if "__pycache__" is mentioned explicitly in the requested pattern. Some examples: base directory: ./ └── __pycache__ └── b.txt >>> _is_inside_unrequested_special_dir("__pycache__/b.txt", "**") True >>> _is_inside_unrequested_special_dir("__pycache__/b.txt", "*/b.txt") True >>> _is_inside_unrequested_special_dir("__pycache__/b.txt", "__pycache__/*") False >>> _is_inside_unrequested_special_dir("__pycache__/b.txt", "__*/*") False cSg|] }|dr|qS__ startswithr=partr&r&r'r?z6_is_inside_unrequested_special_dir..cSrkrlrnrpr&r&r'r?rr)rparentpartsrd)rjr>data_dirs_to_ignore_in_pathdata_dirs_to_ignore_in_patternr&r&r'"_is_inside_unrequested_special_dirsrwcCs8ddt|jD}ddt|jD}t|t|kS)u: When a path matches a pattern, we additionnally check if it's a hidden file or if it's inside a hidden directory we ignore by default, i.e. if the file name or a parent directory name starts with a dot. Users can still explicitly request a filepath that is hidden or is inside a hidden directory if the hidden part is mentioned explicitly in the requested pattern. Some examples: base directory: ./ └── .hidden_file.txt >>> _is_unrequested_hidden_file_or_is_inside_unrequested_hidden_dir(".hidden_file.txt", "**") True >>> _is_unrequested_hidden_file_or_is_inside_unrequested_hidden_dir(".hidden_file.txt", ".*") False base directory: ./ └── .hidden_dir └── a.txt >>> _is_unrequested_hidden_file_or_is_inside_unrequested_hidden_dir(".hidden_dir/a.txt", "**") True >>> _is_unrequested_hidden_file_or_is_inside_unrequested_hidden_dir(".hidden_dir/a.txt", ".*/*") False >>> _is_unrequested_hidden_file_or_is_inside_unrequested_hidden_dir(".hidden_dir/a.txt", ".hidden_dir/*") False base directory: ./ └── .hidden_dir └── .hidden_file.txt >>> _is_unrequested_hidden_file_or_is_inside_unrequested_hidden_dir(".hidden_dir/.hidden_file.txt", "**") True >>> _is_unrequested_hidden_file_or_is_inside_unrequested_hidden_dir(".hidden_dir/.hidden_file.txt", ".*/*") True >>> _is_unrequested_hidden_file_or_is_inside_unrequested_hidden_dir(".hidden_dir/.hidden_file.txt", ".*/.*") False >>> _is_unrequested_hidden_file_or_is_inside_unrequested_hidden_dir(".hidden_dir/.hidden_file.txt", ".hidden_dir/*") True >>> _is_unrequested_hidden_file_or_is_inside_unrequested_hidden_dir(".hidden_dir/.hidden_file.txt", ".hidden_dir/.*") False cS(g|]}|drt|dhks|qS.rorgrpr&r&r'r? zS_is_unrequested_hidden_file_or_is_inside_unrequested_hidden_dir..cSrxryr{rpr&r&r'r?r|)rrtrd)rjr>hidden_directories_in_pathhidden_directories_in_patternr&r&r'?_is_unrequested_hidden_file_or_is_inside_unrequested_hidden_dirs5rpattern_resolverc sFtD]Wdd}z||}Wn tyYqwt|dkrYfdd|DtddDr=tdtd d fd d tDtt t}fd d|DSqt D]<g} D]&\}}|D]}z||}Wn ty{Yqjwt|dkr| |nqjqd|rfdd|DSq\td|d|)a+ Get the default pattern from a directory or repository by testing all the supported patterns. The first patterns to return a non-empty list of data files is returned. In order, it first tests if SPLIT_PATTERN_SHARDED works, otherwise it tests the patterns in ALL_DEFAULT_PATTERNS. z{split}*rcs&h|]}tt|ttdqSr`)rrr)r=p split_patternr&r' sz+_get_data_files_patterns..css|] }tt| VqdSrL)rematchrrDr&r&r'rOsz+_get_data_files_patterns..zSplit name should match 'z '' but got 'z'.csg|] }|vrt|qSr&)rWrD)rir&r'r?z,_get_data_files_patterns..csi|] }|j|dgqS)r`)r:rDrr&r'rFrz,_get_data_files_patterns..csi|]}||qSr&r&rD) patterns_dictr&r'rF+sCouldn't resolve pattern  with resolver ) ALL_SPLIT_PATTERNSreplaceFileNotFoundErrorrdrQrfrDEFAULT_SPLITSsortedrgALL_DEFAULT_PATTERNSrbappend)rr> data_files sorted_splitsnon_empty_splitsrErTr&)rrrir'_get_data_files_patternssH           rc Cs^g}tD]}z||}t|dkr||Wqty Yqw|r%|Std|d|)zM Get the supported metadata patterns from a directory or repository. rrr)METADATA_PATTERNSrdrr)rnon_empty_patternsr>metadata_filesr&r&r'_get_metadata_files_patterns/s   r base_pathallowed_extensionsdownload_configc s~t|r t||}nt|rtj|dtj}nd}t||d\}}t|fi|\}t t t |ht |j tr@|j n|j d}|dkrM|dndi}|dkratjtdkrad|d <fd d |j|fd d i|D}durfdd |D} t| t|krtt |t | } td|d| n|} | sd|d} dur| dt7} t| | S)a{ Resolve the paths and URLs of the data files from the pattern passed by the user. You can use patterns to resolve multiple local files. Here are a few examples: - *.csv to match all the CSV files at the first level - **.csv to match all the CSV files at any level - data/* to match all the files inside "data" - data/** to match all the files inside "data" and its subdirectories The patterns are resolved using the fsspec glob. In fsspec>=2023.12.0 this is equivalent to Python's glob.glob, Path.glob, Path.match and fnmatch where ** is unsupported with a prefix/suffix other than a forward slash /. More generally: - '*' matches any character except a forward-slash (to match just the file or directory name) - '**' matches any character including a forward-slash / Hidden files and directories (i.e. whose names start with a dot) are ignored, unless they are explicitly requested. The same applies to special directories that start with a double underscore like "__pycache__". You can still include one if the pattern explicilty mentions it: - to include a hidden file: "*/.hidden.txt" or "*/.*" - to include a hidden directory: ".hidden/*" or ".*/*" - to include a special directory: "__special__/*" or "__*/*" Example:: >>> from datasets.data_files import resolve_pattern >>> base_path = "." >>> resolve_pattern("docs/**/*.py", base_path) [/Users/mariosasko/Desktop/projects/datasets/docs/source/_config.py'] Args: pattern (str): Unix pattern or paths or URLs of the data files to resolve. The paths can be absolute or relative to base_path. Remote filesystems using fsspec are supported, e.g. with the hf:// protocol. base_path (str): Base path to use when resolving relative paths. allowed_extensions (Optional[list], optional): White-list of file extensions to use. Defaults to None (all extensions). For example: allowed_extensions=[".csv", ".json", ".txt", ".parquet"] Returns: List[str]: List of paths or URLs to the local or remote files that match the patterns. rrfilez://hfz0.20.0F expand_infocsRg|]%\}}|ddkr't|vrt|st|s|r#|n|qS)typer)rrwrro)r=filepathinfo)files_to_ignore fs_patternprotocol_prefixr&r'r?~s  z#resolve_pattern..detailTNcs8g|]}tfddt|dddDr|qS)c3s|] }d|vVqdS)rzNr&)r=suffixrr&r'rOsz-resolve_pattern...rzrN)rQrrE)r=rrr&r'r?s &z Some files matched the pattern 'z-' but don't have valid data file extensions: zUnable to find ''z with any supported extension )rrrosr_ splitdriver8rr rgFILES_TO_IGNORErrXprotocolrWrHF_HUB_VERSIONrparseglobrbrdrYloggerrr) r>rrrstorage_optionsfsr glob_kwargs matched_pathsoutinvalid_matched_files error_msgr&)rrrrr'resolve_pattern@sB/   rcCs:tt||d}zt|WStytd|ddw)uA Get the default pattern from a directory testing all the supported patterns. The first patterns to return a non-empty list of data files is returned. Some examples of supported patterns: Input: my_dataset_repository/ ├── README.md └── dataset.csv Output: {'train': ['**']} Input: my_dataset_repository/ ├── README.md ├── train.csv └── test.csv my_dataset_repository/ ├── README.md └── data/ ├── train.csv └── test.csv my_dataset_repository/ ├── README.md ├── train_0.csv ├── train_1.csv ├── train_2.csv ├── train_3.csv ├── test_0.csv └── test_1.csv Output: {'train': ['**/train[-._ 0-9]*', '**/*[-._ 0-9]train[-._ 0-9]*', '**/training[-._ 0-9]*', '**/*[-._ 0-9]training[-._ 0-9]*'], 'test': ['**/test[-._ 0-9]*', '**/*[-._ 0-9]test[-._ 0-9]*', '**/testing[-._ 0-9]*', '**/*[-._ 0-9]testing[-._ 0-9]*', ...]} Input: my_dataset_repository/ ├── README.md └── data/ ├── train/ │ ├── shard_0.csv │ ├── shard_1.csv │ ├── shard_2.csv │ └── shard_3.csv └── test/ ├── shard_0.csv └── shard_1.csv Output: {'train': ['**/train/**', '**/train[-._ 0-9]*/**', '**/*[-._ 0-9]train/**', '**/*[-._ 0-9]train[-._ 0-9]*/**', ...], 'test': ['**/test/**', '**/test[-._ 0-9]*/**', '**/*[-._ 0-9]test/**', '**/*[-._ 0-9]test[-._ 0-9]*/**', ...]} Input: my_dataset_repository/ ├── README.md └── data/ ├── train-00000-of-00003.csv ├── train-00001-of-00003.csv ├── train-00002-of-00003.csv ├── test-00000-of-00001.csv ├── random-00000-of-00003.csv ├── random-00001-of-00003.csv └── random-00002-of-00003.csv Output: {'train': ['data/train-[0-9][0-9][0-9][0-9][0-9]-of-[0-9][0-9][0-9][0-9][0-9]*.*'], 'test': ['data/test-[0-9][0-9][0-9][0-9][0-9]-of-[0-9][0-9][0-9][0-9][0-9]*.*'], 'random': ['data/random-[0-9][0-9][0-9][0-9][0-9]-of-[0-9][0-9][0-9][0-9][0-9]*.*']} In order, it first tests if SPLIT_PATTERN_SHARDED works, otherwise it tests the patterns in ALL_DEFAULT_PATTERNS. rrThe directory at z doesn't contain any data filesN)rrrrr)rrresolverr&r&r'get_data_patternss T  rcCs:tt||d}zt|WStytd|ddw)zE Get the supported metadata patterns from a local directory. rrz" doesn't contain any metadata fileN)rrrrrr&r&r'get_metadata_patternss   r data_filec Cst||d\}}t|fi|^}}t|tr"||}|j|jfSt|trR|t j rRtt j |j d}d|t t j dd ddd}||}|j|jfS||}dD]}||vrht||fSqYdS) Nr)endpointtokenzhf://rz /resolve/@)ETagetagmtimer&)rr rXr resolve_pathrepo_idrevisionrror HF_ENDPOINTrrdrrrW) rrrr_ resolved_pathhffsrrZr&r&r'_get_single_origin_metadatas    $   rr max_workerscCs:|dur|ntj}ttt|d||tdt|dkpddS)NrzResolving data files)r tqdm_classdescdisable)r&HF_DATASETS_MULTITHREADING_MAX_WORKERSrrrhf_tqdmrd)rrrr&r&r'_get_origin_metadatas rcseZdZdZdeedeeeffdd ZddZe   dd eed e j j d e ed e eed e eddf ddZe   dd eed e ed e eed e eddf ddZe   dd eed e ed e eed e eddf ddZdeeddfddZZS)rUa List of data files (absolute local paths or URLs). It has two construction methods given the user's data files patterns : - ``from_hf_repo``: resolve patterns inside a dataset repository - ``from_local_or_remote``: resolve patterns from a local path Moreover DataFilesList has an additional attribute ``origin_metadata``. It can store: - the last modified time of local files - ETag of remote files - commit sha of a dataset repository Thanks to this additional attribute, it is possible to hash the list and get a different hash if and only if at least one file changed. This is useful for caching Dataset objects that are obtained from a list of data files. rorigin_metadatact|||_dSrL)super__init__r)selfrr __class__r&r'r=s  zDataFilesList.__init__cCtg|||j|jSrL)rUrrotherr&r&r'__add__AzDataFilesList.__add__NrT dataset_inforrrrKcCs6d|jd|jd|p dd}|j||||dS)Nzhf://datasets/r/rrrr)idsharstrip from_patterns)clsrTrrrrr&r&r' from_hf_repoDs$ zDataFilesList.from_hf_repocCs,|dur|nt}|j||||dSNr)rresolveas_posixr)rrTrrrr&r&r'from_local_or_remoteRsz"DataFilesList.from_local_or_remotec Cst|dur|nt}g}|D]}z |t||||dWqty.t|s,Yqwt||d}|||SNrr)rrrextendrrrr)rrTrrrrr>rr&r&r'r_s&   zDataFilesList.from_patterns extensionscsBddd|Dtddtfdd|D|jdS) N|css|]}d|VqdS)\Nr&)r=extr&r&r'rOzrPz2DataFilesList.filter_extensions..z.*(z )(\..+)?$csg|] }|r|qSr&)r)r=rrNr&r'r?}rrz3DataFilesList.filter_extensions..)r)joinrcompilerUrrrr&rNr'filter_extensionsys zDataFilesList.filter_extensionsNNN)r#r$r%__doc__rrWr rr classmethodhuggingface_hubhf_api DatasetInfor rrrrr __classcell__r&r&rr'rU+sh"        c@seZdZdZe   ddeeeeee ffde ede eede e ddf dd Z e   ddeeeeee ffd e jjde ede eede e ddf d d Ze   ddeeeeee ffde ede eede e ddf d dZdeeddfddZdS) DataFilesDicta Dict of split_name -> list of data files (absolute local paths or URLs). It has two construction methods given the user's data files patterns : - ``from_hf_repo``: resolve patterns inside a dataset repository - ``from_local_or_remote``: resolve patterns from a local path Moreover each list is a DataFilesList. It is possible to hash the dictionary and get a different hash if and only if at least one file changed. For more info, see ``DataFilesList``. This is useful for caching Dataset objects that are obtained from a list of data files. Changing the order of the keys of this dictionary also doesn't change its hash. NrTrrrrKcC@|}|D]\}}t|tstj||||dn|||<q|Sr)rbrXrUrrrTrrrrrZpatterns_for_keyr&r&r'r z"DataFilesDict.from_local_or_remoterc CsB|}|D]\}}t|tstj|||||dn|||<q|S)N)rrrr)rbrXrUr) rrTrrrrrrZrr&r&r'rs   zDataFilesDict.from_hf_repocCrr)rbrXrUrrr&r&r'rrzDataFilesDict.from_patternsrcC.t|}|D] \}}||||<q |SrLrrbr)rrrrZdata_files_listr&r&r'r zDataFilesDict.filter_extensionsr)r#r$r%rrrrWr rrUr rrrrrrrrr&r&r&r'rsd      rcseZdZdZdeedeeeeffdd ZddZe ddeedeeed d fd d Z dd edee d dfddZ deed dfddZ ZS)DataFilesPatternsListz List of data files patterns (absolute local paths or URLs). For each pattern there should also be a list of allowed extensions to keep, or a None ot keep all the files for the pattern. rTrcrrL)rrr)rrTrrr&r'rs  zDataFilesPatternsList.__init__cCrrL)rUrrr&r&r'rrzDataFilesPatternsList.__add__NrKDataFilesPatternsDictcCs|||gt|SrL)rd)rrTrr&r&r'rsz#DataFilesPatternsList.from_patternsrrrUc Cs|dur|nt}g}t||jD]\}}z |t||||dWqty4t|s2Yqwt ||d}t ||Sr) rrrziprrrrrrrU)rrrrr>rrr&r&r'rs&   zDataFilesPatternsList.resolvercst|fdd|jDS)Ncsg|]}|qSr&r&)r=rrr&r'r?raz;DataFilesPatternsList.filter_extensions..)r rrr&rr'r sz'DataFilesPatternsList.filter_extensionsrL)r#r$r%rrrWr rrrrrrrrr&r&rr'r s4   r c@sveZdZdZe ddeeeefdeeeddfddZ dded ee dd fd d Z d eeddfddZ dS)rz[ Dict of split_name -> list of data files patterns (absolute local paths or URLs). NrTrrKcCs<|}|D]\}}t|tstj||dn|||<q|S)Nr)rbrXr r)rrTrrrZrr&r&r'rsz#DataFilesPatternsDict.from_patternsrrrcCs,t}|D] \}}|||||<q|SrL)rrbr)rrrrrZdata_files_patterns_listr&r&r'r'szDataFilesPatternsDict.resolvercCr rLr )rrrrZrr&r&r'r1r z'DataFilesPatternsDict.filter_extensionsrL) r#r$r%rrrrWrr rrrrr&r&r&r'rs(    r)NNrL)\rr functoolsrrrpathlibrrtypingrrrr r r r r fsspec.corer fsspec.implementations.httprr packagingrtqdm.contrib.concurrentrrrdownloadrnamingrrirutilsrrrutils.file_utilsrrrrrutils.py_utilsrrrWTRAINrc get_loggerr#rr!rr)SPLIT_PATTERN_SHARDED VALIDATIONTESTrCr;FSSPEC_VERSIONrr9rHr"DEFAULT_PATTERNS_SPLIT_IN_FILENAME"DEFAULT_PATTERNS_SPLIT_IN_DIR_NAMEDEFAULT_PATTERNS_ALLrrrrRrboolrSrhrwrrrrrrrintrrUrr rr&r&r&r's  $                     0&!,>$*  ([]   W[5