o +ifdj @sFdZddlZddlZddlZddlZddlZddlmZddlm Z ddl m Z ddl m Z mZmZmZddlZddlmZddlmZdd lmZdd lmZmZdd lmZmZm Z m!Z!dd l"m#Z#dd l"m$Z%ddl&m'Z'ddl(m)Z)e rddl*m+Z+ddl,m-Z-m.Z.e#/e0Z1dZ2e3dedddej4Z5e3dZ6Gddde7Z8Gddde7Z9eddGdddZ:  dGd-d.d.e=Z?eddGd/d0d0Z@eddGd1d2d2ZAd3d4ZBd5d6ZCd7d8ZDd9d:ZEGd;d#d#ZFdS)=zArrow ArrowReader.N) dataclass)partial)Path) TYPE_CHECKINGListOptionalUnion) thread_map)DownloadConfig) _split_refilenames_for_dataset_split) InMemoryTableMemoryMappedTableTable concat_tables)logging)tqdm) deprecated) cached_path) DatasetInfo)Split SplitInfoz=https://storage.googleapis.com/huggingface-nlp/cache/datasetsz ^ (?Pz) (\[ ((?P-?\d+) (?P%)?)? : ((?P-?\d+) (?P%)?)? \])?(\((?P[^\)]*)\))? $ z\s*\+\s*c@eZdZdZdS)DatasetNotOnHfGcsErrorz?When you can't get the dataset from the Hf google cloud storageN__name__ __module__ __qualname____doc__r!r!N/var/www/html/corbot_env/lib/python3.10/site-packages/datasets/arrow_reader.pyrErc@r)MissingFilesOnHfGcsErrorz9When some files are missing on the Hf oogle cloud storageNrr!r!r!r"r$Kr#r$T)frozenc@s&eZdZUdZeed<eeed<dS)FileInstructionsa}The file instructions associated with a split ReadInstruction. Attributes: num_examples: `int`, The total number of examples file_instructions: List[dict(filename, skip, take)], the files information. The filenames contains the relative path, not absolute. skip/take indicates which example read in the file: `ds.slice(skip, take)` num_examplesfile_instructionsN)rrrr int__annotations__rdictr!r!r!r"r&Qs  r&name split_infosr instructionReadInstructionfiletype_suffix prefix_pathreturncsttstdtjstddd|D}dd|Dfdd|D}t|ts9t|}||}g}d} |D]} || j } || j } | j } | j dur\dn| j }| j durf| n| j }| dur| D]}||}|dkrzqo| |7} | |||d qoqDd}d}t | | D]D\}}||7}||kr||kr||kr||nd}||kr|||nd }|dkrq| |||d | |d kr||n|7} ||7}qqDt| |d S) aReturns instructions of the split dict. Args: name (`str`): Name of the dataset. split_infos (`list` of `[SplitInfo]`): Dataset splits information. instruction ([`ReadInstruction`] or `str`): Reading instruction for a dataset. filetype_suffix (`str`, *optional*): Suffix of dataset files, e.g. 'arrow' or 'parquet'. prefix_path (`str`, *optional*): Prefix of dataset files, e.g. directory name. Returns: [`FileInstructions`] zExpected str 'name', but got: zExpected non-empty str 'name'cSi|]}|j|jqSr!)r,r'.0infor!r!r" wz*make_file_instructions..cSr3r!)r, shard_lengthsr4r!r!r"r7xr8c s*i|]}|jt|j|jdqS))path dataset_namesplitr0r9)r,r r4r0r,name2shard_lengthsr1r!r"r7ysrN)filenameskiptaker)r'r() isinstancestr TypeErrortyper ValueErrorr/ from_spec to_absolute splitnamefrom_toappendzipr&)r,r-r.r0r1name2lenname2filenamesabsolute_instructionsr(r' abs_instr split_length filenamesr9rJrKr?rA index_start index_end shard_lengthr@r!r=r"make_file_instructions`sZ        rWc@seZdZdZdededfddZddefd d Zddefd d Z d dZ dddZ  dde e dedfddZedefddZdS) BaseReaderz@ Build a Dataset object out of Instruction instance(s). r:r6rcCs||_||_d|_dS)zInitializes ArrowReader. Args: path (str): path where tfrecords are stored. info (DatasetInfo): info about the dataset. N)_path_info_filetype_suffixselfr:r6r!r!r"__init__s zBaseReader.__init__Fr2cCst)=Returns a Dataset instance from given (filename, skip, take).)NotImplementedError)r]filename_skip_take in_memoryr!r!r"_get_table_from_filenamesz#BaseReader._get_table_from_filenamecCst|dkstdd|Dstdt|}|D]}tj|j|d|d<qt t |j |d|t dt|dkp:d d }d d |D}|sU|j d usQ|j jd urUtd |pdtjgt|j jjdg}t|dkrqt|}|S|d}|S)aReturns Dataset for given file instructions. Args: files: List[dict(filename, skip, take)], the files information. The filenames contain the absolute path, not relative. skip/take indicates which example read in the file: `ds.slice(skip, take)` in_memory (bool, default False): Whether to copy the data in-memory. rcss|]}t|tVqdSN)rBr+)r5fr!r!r" sz)BaseReader._read_files..z&please provide valid file informationsr?rbzLoading dataset shardsN) tqdm_classdescdisablecSsg|] }t|dkr|qS)r)len)r5tr!r!r" sz*BaseReader._read_files..zqTried to read an empty table. Please specify at least info.features to create an empty table with the right type.)schemar )rlallrFcopydeepcopyosr:joinrYr rrchf_tqdmrZfeaturesr from_batchesparorEr)r]filesrbre pa_tablespa_tabler!r!r" _read_filess,    zBaseReader._read_filescCs t||||j|jd}|j}|S)z?Return list of dict {'filename': str, 'skip': int, 'take': int})r0r1)rWr[rYr()r]r,r.r-r(ryr!r!r"get_file_instructionss z BaseReader.get_file_instructionscCs6||||}|sd|d}t||j|||dS)a Returns Dataset instance(s). Args: name (str): name of the dataset. instructions (ReadInstruction): instructions to read. Instruction can be string and will then be passed to the Instruction constructor as it. split_infos (list of SplitInfo proto): the available splits for dataset. in_memory (bool, default False): Whether to copy the data in-memory. Returns: kwargs to build a single Dataset instance. z Instruction "z" corresponds to no data!)ryoriginal_instructionsrb)r}rF read_files)r]r, instructionsr-rbrymsgr!r!r"reads  zBaseReader.readNryr~)Nr/rcCsF|j||d}|durddlm}|t|}nd}||j|d}|S)aJReturns single Dataset instance for the set of file instructions. Args: files: List[dict(filename, skip, take)], the files information. The filenames contains the relative path, not absolute. skip/take indicates which example read in the file: `ds.skip().take()` original_instructions: store the original instructions used to build the dataset split in the dataset. in_memory (bool, default False): Whether to copy the data in-memory. Returns: kwargs to build a Dataset instance. rgNr )r) arrow_tabler6r<)r|splitsrrCrZ)r]ryr~rbr{rr<dataset_kwargsr!r!r"rs zBaseReader.read_filesdownload_configc CsDtd|tjd}z0tj|d}t|tjd|d}t|tj|j d|j dur:|j |j |j Wnt yL}zt|dd}~wwzC|j jD];}|j|j j||j jd}|D](} tt| d|j } tj|| } t| tjd|d} t| | dqdqRWdSt y}zt|dd}~ww)a% Download the dataset files from the Hf GCS Args: dl_cache_dir: `str`, the local cache directory used to download files relative_data_dir: `str`, the relative directory of the remote files from the `datasets` directory on GCS. /zdataset_info.json)rN)r,r.r-r?)HF_GCP_BASE_URLreplacerssepr:rtrshutilmoverYrZupdatefrom_directoryFileNotFoundErrorrrr} builder_namevaluesrCr relative_tor$) r]rrelative_data_dirremote_cache_dirremote_dataset_infodownloaded_dataset_infoerrr<r(file_instructionfile_to_downloadremote_prepared_filenamedownloaded_prepared_filenamer!r!r"download_from_hf_gcs sD      zBaseReader.download_from_hf_gcsF)NF)rrrr rCrr^rrcr|r}rrr+rrrr rr!r!r!r"rXs"    rXcsReZdZdZdededffdd Zd defd d Ze d defd d Z Z S) ArrowReaderz Build a Dataset object out of Instruction instance(s). This Reader uses either memory mapping or file descriptors (in-memory) on arrow files. r:r6rct||d|_dS)zInitializes ArrowReader. Args: path (str): path where Arrow files are stored. info (DatasetInfo): info about the dataset. arrowNsuperr^r[r\ __class__r!r"r^N zArrowReader.__init__Fr2cCs|dd|vr |dndd|vr|dnd}}}tj||d}|dkr+t||}|durC|durC|dkr=|t|ksC|||}|S)r_r?r@NrArgrr)r read_tablerlslice)r]rarbr?r@rAtabler!r!r"rcXs  $ z$ArrowReader._get_table_from_filenamecCs|rtnt}||S)z Read table from file. Args: filename (str): File name of the table. in_memory (bool, default=False): Whether to copy the data in-memory. Returns: pyarrow.Table )rr from_file)r?rb table_clsr!r!r"rgs zArrowReader.read_tabler) rrrr rCrr^rrc staticmethodr __classcell__r!r!rr"rHs  rcs6eZdZdZdededffdd ZddZZS) ParquetReaderzv Build a Dataset object out of Instruction instance(s). This Reader uses memory mapping on parquet files. r:r6rcr)zInitializes ParquetReader. Args: path (str): path where tfrecords are stored. info (DatasetInfo): info about the dataset. parquetNrr\rr!r"r^}rzParquetReader.__init__cKsv|dd|vr |dndd|vr|dnd}}}tj|dd}|dur9|dur9|dkr3|t|ks9|||}|S)r_r?r@NrAT) memory_mapr)pqrrlr)r]rakwargsr?r@rAr{r!r!r"rcs $ z&ParquetReader._get_table_from_filename) rrrr rCrr^rcrr!r!rr"rws rc@s*eZdZUdZeed<eed<eed<dS)_AbsoluteInstructionz?A machine friendly slice: defined absolute positive boundaries.rIrJrKN)rrrr rCr*r)r!r!r!r"rs  rc@sbeZdZUdZeed<dZeeed<dZ eeed<dZ eeed<dZ eeed<dd Z dS) _RelativeInstructionzHRepresents a single parsed slicing instruction, can use % and negatives.rINrJrKunitroundingcCs|jdur|jdvrtd|jdur|jdvrtd|jdkr*|jdur*td|jdkr?|jdur?t|jdkr?td|jdkrT|jdurTt|jdkrTtd|jdur`|jdkr`d n|j|jd <dS) N)%abszunit must be either % or abs)closestpct1_dropremainderz5rounding must be either closest or pct1_dropremainderrzAIt is forbidden to specify rounding if not using percent slicing.dz2Percent slice boundaries must be > -100 and < 100.rr)rrFrrJrrK__dict__r]r!r!r" __post_init__s""(z"_RelativeInstruction.__post_init__) rrrr rCr*rJrr)rKrrrr!r!r!r"rs  rcCst|}|std||ds|drdnd}t|d|d|dr1t|dnd |d rAt|d |d Sd |d S) z)Returns ReadInstruction for given string.z!Unrecognized instruction format: from_pctto_pctrrr<rfromNrK) split_namerrJrKr) _SUB_SPEC_REmatchrFgroupr/r))specresrr!r!r"_str_to_read_instructions rcCs&|dkr d}t||t|dS)NrzUsing "pct1_dropremainder" rounding on a split with less than 100 elements is forbidden: it always results in an empty dataset.Y@)rFmathtrunc)boundaryr'rr!r!r"_pct_to_abs_pct1s rcCstt||dS)Nr)r)round)rr'r!r!r"_pct_to_abs_closestsrcCs|jdkrtnt}|j}||vrtd|dt|d||}|j}|j}|jdkrC|dur2dn|||}|dur=|n|||}n|durIdn|}|durQ|n|}|dkr^t ||d}|dkrit ||d}t ||}t ||}t |||S)zReturns _AbsoluteInstruction instance for given RelativeInstruction. Args: rel_instr: RelativeInstruction instance. name2len: dict {split_name: num_examples}. rzUnknown split "z". Should be one of .rNr) rrrrIrFlistrJrKrmaxminr) rel_instrrN pct_to_absr<r'rJrKr!r!r"_rel_to_abs_instrs&    rc@sbeZdZdZddZeddZdddZed d Zd d Z d dZ ddZ ddZ ddZ dS)r/aReading instruction for a dataset. Examples:: # The following lines are equivalent: ds = datasets.load_dataset('mnist', split='test[:33%]') ds = datasets.load_dataset('mnist', split=datasets.ReadInstruction.from_spec('test[:33%]')) ds = datasets.load_dataset('mnist', split=datasets.ReadInstruction('test', to=33, unit='%')) ds = datasets.load_dataset('mnist', split=datasets.ReadInstruction( 'test', from_=0, to=33, unit='%')) # The following lines are equivalent: ds = datasets.load_dataset('mnist', split='test[:33%]+train[1:-1]') ds = datasets.load_dataset('mnist', split=datasets.ReadInstruction.from_spec( 'test[:33%]+train[1:-1]')) ds = datasets.load_dataset('mnist', split=( datasets.ReadInstruction('test', to=33, unit='%') + datasets.ReadInstruction('train', from_=1, to=-1, unit='abs'))) # The following lines are equivalent: ds = datasets.load_dataset('mnist', split='test[:33%](pct1_dropremainder)') ds = datasets.load_dataset('mnist', split=datasets.ReadInstruction.from_spec( 'test[:33%](pct1_dropremainder)')) ds = datasets.load_dataset('mnist', split=datasets.ReadInstruction( 'test', from_=0, to=33, unit='%', rounding="pct1_dropremainder")) # 10-fold validation: tests = datasets.load_dataset( 'mnist', [datasets.ReadInstruction('train', from_=k, to=k+10, unit='%') for k in range(0, 100, 10)]) trains = datasets.load_dataset( 'mnist', [datasets.ReadInstruction('train', to=k, unit='%') + datasets.ReadInstruction('train', from_=k+10, unit='%') for k in range(0, 100, 10)]) cCs ||_dSrd_relative_instructions)r]relative_instructionsr!r!r"_inits zReadInstruction._initcCs||}|||S)zCReturns ReadInstruction obj initialized with relative_instructions.)__new__r)clsrresultr!r!r",_read_instruction_from_relative_instructionss  z.r N)rC_ADDITION_SEP_REr<rFrsum)rrsubsr.r!r!r"rGCs   zReadInstruction.from_specc Csg}|jD]c}|j}|jdus|jdurc|j}|j}|j}|j}|dkr&|nd}|dur2t||nd}|dur>t||nd}d|d|d}|dkr[|dur[|dkr[d|dnd} ||| 7}||qd |S) Nr[:]r()+) rrIrJrKrrrCrLrt) r]rel_instr_specsrrel_instr_specrJrKrr slice_str rounding_strr!r!r"to_specbs" &   zReadInstruction.to_speccCsjt|ts d}t||j}|j}|djdkr.|djdkr.|jdj|djkr.td|||S)zEReturns a new ReadInstruction obj, result of appending other to self.zAReadInstruction can only be added to another ReadInstruction obj.rrzPIt is forbidden to sum ReadInstruction instances with different rounding values.)rBr/rDrrrrFr)r]otherrself_ris other_risr!r!r"__add__vs zReadInstruction.__add__cCs|Srd)rrr!r!r"__str__szReadInstruction.__str__cCsd|jdS)NzReadInstruction(rrrr!r!r"__repr__szReadInstruction.__repr__csfdd|jDS)aZTranslate instruction into a list of absolute instructions. Those absolute instructions are then to be added together. Args: name2len (`dict`): Associating split names to number of examples. Returns: list of _AbsoluteInstruction instances (corresponds to the + in spec). csg|]}t|qSr!)r)r5rrNr!r"rnr8z/ReadInstruction.to_absolute..r)r]rNr!rr"rHs zReadInstruction.to_absolute)NNNN)rrrr r classmethodrr^rGrrrrrHr!r!r!r"r/s&    )NN)Gr rqrrsrer dataclassesr functoolsrpathlibrtypingrrrrpyarrowrxpyarrow.parquetrrtqdm.contrib.concurrentr download.download_configr namingr r rrrrrutilsrrruutils.deprecation_utilsrutils.file_utilsrr6rrrr get_loggerrloggerrcompileXrrConnectionErrorrr$r&rCrWrXrrrrrrrrr/r!r!r!r"s                 K/