o TZh"j @s:dZddlZddlZddlZddlZddlZddlmZddlm Z ddl m Z ddl m Z mZmZmZddlZddlmZddlmZdd lmZdd lmZmZdd lmZmZm Z m!Z!dd l"m#Z#dd l"m$Z%ddl&m'Z'e rddl(m)Z)ddl*m+Z+m,Z,e#-e.Z/dZ0e1dedddej2Z3e1dZ4Gddde5Z6Gddde5Z7eddGdddZ8  d;de9ded d!ee9d"fd#ee9d$ee9d%e8f d&d'Z:Gd(d)d)Z;Gd*d+d+e;ZeddGd0d1d1Z?d2d3Z@d4d5ZAd6d7ZBd8d9ZCGd:d"d"ZDdS)z) (\[ ((?P-?\d+) (?P%)?)? : ((?P-?\d+) (?P%)?)? \])?(\((?P[^\)]*)\))? $ z\s*\+\s*c@eZdZdZdS)DatasetNotOnHfGcsErrorz?When you can't get the dataset from the Hf google cloud storageN__name__ __module__ __qualname____doc__r r L/var/www/html/lang_env/lib/python3.10/site-packages/datasets/arrow_reader.pyrDrc@r)MissingFilesOnHfGcsErrorz9When some files are missing on the Hf oogle cloud storageNrr r r r!r#Jr"r#T)frozenc@s&eZdZUdZeed<eeed<dS)FileInstructionsa}The file instructions associated with a split ReadInstruction. Attributes: num_examples: `int`, The total number of examples file_instructions: List[dict(filename, skip, take)], the files information. The filenames contains the relative path, not absolute. skip/take indicates which example read in the file: `ds.slice(skip, take)` num_examplesfile_instructionsN)rrrrint__annotations__rdictr r r r!r%Ps  r%name split_infosr instructionReadInstructionfiletype_suffix prefix_pathreturncsttstdtjstddd|D}dd|Dfdd|D}t|ts9t|}||}g}d} |D]} || j } || j } | j } | j dur\dn| j }| j durf| n| j }| dur| D]}||}|dkrzqo| |7} | |||d qoqDd}d}t | | D]D\}}||7}||kr||kr||kr||nd}||kr|||nd }|dkrq| |||d | |d kr||n|7} ||7}qqDt| |d S) aReturns instructions of the split dict. Args: name (`str`): Name of the dataset. split_infos (`list` of `[SplitInfo]`): Dataset splits information. instruction ([`ReadInstruction`] or `str`): Reading instruction for a dataset. filetype_suffix (`str`, *optional*): Suffix of dataset files, e.g. 'arrow' or 'parquet'. prefix_path (`str`, *optional*): Prefix of dataset files, e.g. directory name. Returns: [`FileInstructions`] zExpected str 'name', but got: zExpected non-empty str 'name'cSi|]}|j|jqSr )r+r&.0infor r r! vz*make_file_instructions..cSr2r )r+ shard_lengthsr3r r r!r6wr7c s*i|]}|jt|j|jdqS))pathZ dataset_namesplitr/r8)r+r r3r/r+Zname2shard_lengthsr0r r!r6xsrN)filenameskiptaker)r&r') isinstancestr TypeErrortyper ValueErrorr. from_spec to_absolute splitnamefrom_toappendzipr%)r+r,r-r/r0name2lenZname2filenamesZabsolute_instructionsr'r&Z abs_instrZ split_length filenamesr8rGrHr<r>Z index_startZ index_endZ shard_lengthr=r r;r!make_file_instructions_sZ        rMc@seZdZdZdededfddZddefd d Zddefd d Z d dZ dddZ  dde e dedfddZdefddZdS) BaseReaderz@ Build a Dataset object out of Instruction instance(s). r9r5rcCs||_||_d|_dS)zInitializes ArrowReader. Args: path (str): path where tfrecords are stored. info (DatasetInfo): info about the dataset. N)_path_info_filetype_suffixselfr9r5r r r!__init__s zBaseReader.__init__Fr1cCst)=Returns a Dataset instance from given (filename, skip, take).)NotImplementedError)rSfilename_skip_take in_memoryr r r!_get_table_from_filenamesz#BaseReader._get_table_from_filenamecCst|dkstdd|Dstdt|}|D]}tj|j|d|d<qt t |j |d|t dt|dkp:d d }d d |D}|sU|j d usQ|j jd urUtd |pdtjgt|j jjdg}t|dkrqt|}|S|d}|S)aReturns Dataset for given file instructions. Args: files: List[dict(filename, skip, take)], the files information. The filenames contain the absolute path, not relative. skip/take indicates which example read in the file: `ds.slice(skip, take)` in_memory (bool, default False): Whether to copy the data in-memory. rcss|]}t|tVqdSN)r?r*)r4fr r r! sz)BaseReader._read_files..z&please provide valid file informationsr<rXzLoading dataset shardsN)Z tqdm_classZdescdisablecSsg|] }t|dkr|qS)r)len)r4tr r r! sz*BaseReader._read_files..zqTried to read an empty table. Please specify at least info.features to create an empty table with the right type.)schemar )r`allrCcopydeepcopyosr9joinrOr rrYhf_tqdmrPfeaturesrZ from_batchesparcrBr)rSfilesrXr[Z pa_tablespa_tabler r r! _read_filess,    zBaseReader._read_filescCs t||||j|jd}|j}|S)z?Return list of dict {'filename': str, 'skip': int, 'take': int})r/r0)rMrQrOr')rSr+r-r,r'rlr r r!get_file_instructionss z BaseReader.get_file_instructionscCs6||||}|sd|d}t||j|||dS)a Returns Dataset instance(s). Args: name (str): name of the dataset. instructions (ReadInstruction): instructions to read. Instruction can be string and will then be passed to the Instruction constructor as it. split_infos (list of SplitInfo proto): the available splits for dataset. in_memory (bool, default False): Whether to copy the data in-memory. Returns: kwargs to build a single Dataset instance. z Instruction "z" corresponds to no data!)rloriginal_instructionsrX)rorC read_files)rSr+Z instructionsr,rXrlmsgr r r!reads  zBaseReader.readNrlrp)Nr.rcCsF|j||d}|durddlm}|t|}nd}||j|d}|S)aJReturns single Dataset instance for the set of file instructions. Args: files: List[dict(filename, skip, take)], the files information. The filenames contains the relative path, not absolute. skip/take indicates which example read in the file: `ds.skip().take()` original_instructions: store the original instructions used to build the dataset split in the dataset. in_memory (bool, default False): Whether to copy the data in-memory. Returns: kwargs to build a Dataset instance. r]Nr )r)Z arrow_tabler5r:)rnsplitsrr@rP)rSrlrprXrmrr:Zdataset_kwargsr r r!rqs zBaseReader.read_filesdownload_configc CsDtd|tjd}z0tj|d}t|tjd|d}t|tj|j d|j dur:|j |j |j Wnt yL}zt|dd}~wwzC|j jD];}|j|j j||j jd}|D](} tt| d|j } tj|| } t| tjd|d} t| | dqdqRWdSt y}zt|dd}~ww)a% Download the dataset files from the Hf GCS Args: dl_cache_dir: `str`, the local cache directory used to download files relative_data_dir: `str`, the relative directory of the remote files from the `datasets` directory on GCS. /zdataset_info.json)ruN)r+r-r,r<)HF_GCP_BASE_URLreplacergsepr9rhrshutilmoverOrPupdatefrom_directoryFileNotFoundErrorrrtroZ builder_namevaluesr@r relative_tor#) rSruZrelative_data_dirZremote_cache_dirZremote_dataset_infoZdownloaded_dataset_infoerrr:r'Zfile_instructionZfile_to_downloadZremote_prepared_filenameZdownloaded_prepared_filenamer r r!download_from_hf_gcssD      zBaseReader.download_from_hf_gcsF)NF)rrrrr@rrTrrYrnrorsrr*rrqr rr r r r!rNs     rNcsReZdZdZdededffdd Zd defd d Ze d defd d Z Z S) ArrowReaderz Build a Dataset object out of Instruction instance(s). This Reader uses either memory mapping or file descriptors (in-memory) on arrow files. r9r5rct||d|_dS)zInitializes ArrowReader. Args: path (str): path where Arrow files are stored. info (DatasetInfo): info about the dataset. arrowNsuperrTrQrR __class__r r!rTL zArrowReader.__init__Fr1cCs|dd|vr |dndd|vr|dnd}}}tj||d}|dkr+t||}|durC|durC|dkr=|t|ksC|||}|S)rUr<r=Nr>r]rr)r read_tabler`slice)rSrWrXr<r=r>tabler r r!rYVs  $ z$ArrowReader._get_table_from_filenamecCs|rtnt}||S)z Read table from file. Args: filename (str): File name of the table. in_memory (bool, default=False): Whether to copy the data in-memory. Returns: pyarrow.Table )rr from_file)r<rXZ table_clsr r r!res zArrowReader.read_tabler) rrrrr@rrTrrY staticmethodr __classcell__r r rr!rFs  rcs6eZdZdZdededffdd ZddZZS) ParquetReaderzv Build a Dataset object out of Instruction instance(s). This Reader uses memory mapping on parquet files. r9r5rcr)zInitializes ParquetReader. Args: path (str): path where tfrecords are stored. info (DatasetInfo): info about the dataset. parquetNrrRrr r!rT{rzParquetReader.__init__cKsv|dd|vr |dndd|vr|dnd}}}tj|dd}|dur9|dur9|dkr3|t|ks9|||}|S)rUr<r=Nr>T)Z memory_mapr)pqrr`r)rSrWkwargsr<r=r>rmr r r!rYs $ z&ParquetReader._get_table_from_filename) rrrrr@rrTrYrr r rr!rus rc@s*eZdZUdZeed<eed<eed<dS)_AbsoluteInstructionz?A machine friendly slice: defined absolute positive boundaries.rFrGrHN)rrrrr@r)r(r r r r!rs  rc@sbeZdZUdZeed<dZeeed<dZ eeed<dZ eeed<dZ eeed<dd Z dS) _RelativeInstructionzHRepresents a single parsed slicing instruction, can use % and negatives.rFNrGrHunitroundingcCs|jdur|jdvrtd|jdur|jdvrtd|jdkr*|jdur*td|jdkr?|jdur?t|jdkr?td|jdkrT|jdurTt|jdkrTtd|jdur`|jdkr`d n|j|jd <dS) N)%abszunit must be either % or abs)closestZpct1_dropremainderz5rounding must be either closest or pct1_dropremainderrzAIt is forbidden to specify rounding if not using percent slicing.dz2Percent slice boundaries must be > -100 and < 100.rr)rrCrrGrrH__dict__rSr r r! __post_init__s""(z"_RelativeInstruction.__post_init__) rrrrr@r)rGrr(rHrrrr r r r!rs  rcCst|}|std||ds|drdnd}t|d|d|dr1t|dnd |d rAt|d |d Sd |d S) z)Returns ReadInstruction for given string.z!Unrecognized instruction format: Zfrom_pctZto_pctrrr:rfromNrH) split_namerrGrHr) _SUB_SPEC_REmatchrCgroupr.r()specresrr r r!_str_to_read_instructions rcCs&|dkr d}t||t|dS)NrzUsing "pct1_dropremainder" rounding on a split with less than 100 elements is forbidden: it always results in an empty dataset.Y@)rCmathtrunc)boundaryr&rrr r r!_pct_to_abs_pct1s rcCstt||dS)Nr)r(round)rr&r r r!_pct_to_abs_closestsrcCs|jdkrtnt}|j}||vrtd|dt|d||}|j}|j}|jdkrC|dur2dn|||}|dur=|n|||}n|durIdn|}|durQ|n|}|dkr^t ||d}|dkrit ||d}t ||}t ||}t |||S)zReturns _AbsoluteInstruction instance for given RelativeInstruction. Args: rel_instr: RelativeInstruction instance. name2len: dict {split_name: num_examples}. rzUnknown split "z". Should be one of .rNr) rrrrFrClistrGrHrmaxminr) rel_instrrKZ pct_to_absr:r&rGrHr r r!_rel_to_abs_instrs&    rc@sbeZdZdZddZeddZdddZed d Zd d Z d dZ ddZ ddZ ddZ dS)r.aReading instruction for a dataset. Examples:: # The following lines are equivalent: ds = datasets.load_dataset('mnist', split='test[:33%]') ds = datasets.load_dataset('mnist', split=datasets.ReadInstruction.from_spec('test[:33%]')) ds = datasets.load_dataset('mnist', split=datasets.ReadInstruction('test', to=33, unit='%')) ds = datasets.load_dataset('mnist', split=datasets.ReadInstruction( 'test', from_=0, to=33, unit='%')) # The following lines are equivalent: ds = datasets.load_dataset('mnist', split='test[:33%]+train[1:-1]') ds = datasets.load_dataset('mnist', split=datasets.ReadInstruction.from_spec( 'test[:33%]+train[1:-1]')) ds = datasets.load_dataset('mnist', split=( datasets.ReadInstruction('test', to=33, unit='%') + datasets.ReadInstruction('train', from_=1, to=-1, unit='abs'))) # The following lines are equivalent: ds = datasets.load_dataset('mnist', split='test[:33%](pct1_dropremainder)') ds = datasets.load_dataset('mnist', split=datasets.ReadInstruction.from_spec( 'test[:33%](pct1_dropremainder)')) ds = datasets.load_dataset('mnist', split=datasets.ReadInstruction( 'test', from_=0, to=33, unit='%', rounding="pct1_dropremainder")) # 10-fold validation: tests = datasets.load_dataset( 'mnist', [datasets.ReadInstruction('train', from_=k, to=k+10, unit='%') for k in range(0, 100, 10)]) trains = datasets.load_dataset( 'mnist', [datasets.ReadInstruction('train', to=k, unit='%') + datasets.ReadInstruction('train', from_=k+10, unit='%') for k in range(0, 100, 10)]) cCs ||_dSrZ_relative_instructions)rSrelative_instructionsr r r!_inits zReadInstruction._initcCs||}|||S)zCReturns ReadInstruction obj initialized with relative_instructions.)__new__r)clsrresultr r r!,_read_instruction_from_relative_instructionss  z.r N)r@_ADDITION_SEP_REr:rCrsum)rrsubsr-r r r!rDAs   zReadInstruction.from_specc Csg}|jD]c}|j}|jdus|jdurc|j}|j}|j}|j}|dkr&|nd}|dur2t||nd}|dur>t||nd}d|d|d}|dkr[|dur[|dkr[d|dnd} ||| 7}||qd |S) Nr[:]r()+) rrFrGrHrrr@rIrh) rSZrel_instr_specsrZrel_instr_specrGrHrrZ slice_strZ rounding_strr r r!to_spec`s" &   zReadInstruction.to_speccCsjt|ts d}t||j}|j}|djdkr.|djdkr.|jdj|djkr.td|||S)zEReturns a new ReadInstruction obj, result of appending other to self.zAReadInstruction can only be added to another ReadInstruction obj.rrzPIt is forbidden to sum ReadInstruction instances with different rounding values.)r?r.rArrrrCr)rSotherrrZself_risZ other_risr r r!__add__ts zReadInstruction.__add__cCs|SrZ)rrr r r!__str__szReadInstruction.__str__cCsd|jdS)NzReadInstruction(rrrr r r!__repr__szReadInstruction.__repr__csfdd|jDS)aZTranslate instruction into a list of absolute instructions. Those absolute instructions are then to be added together. Args: name2len (`dict`): Associating split names to number of examples. Returns: list of _AbsoluteInstruction instances (corresponds to the + in spec). csg|]}t|qSr )r)r4rrKr r!rbr7z/ReadInstruction.to_absolute..r)rSrKr rr!rEs zReadInstruction.to_absolute)NNNN)rrrrr classmethodrrTrDrrrrrEr r r r!r.s&    )NN)Errerrgrerz dataclassesr functoolsrpathlibrtypingrrrrZpyarrowrkZpyarrow.parquetrrZtqdm.contrib.concurrentr Zdownload.download_configr Znamingr r rrrrrutilsrrriZutils.file_utilsrr5rrtrrZ get_loggerrloggerrwcompileXrrConnectionErrorrr#r%r@rMrNrrrrrrrrr.r r r r!s~                K/