o TZhm@sBdZddlZddlZddlZddlmZmZmZmZm Z m Z ddl Z ddl ZddlmZmZddlmZddlmZddlmZdd lmZdd lmZdd lmZdd lmZdd l m!Z!m"Z"ddl#m$Z$ddl%m&Z&ddl'm(Z(ddl)m*Z*ddl+m,Z,m-Z-e*e.Z/GdddeZ0ddZ1GdddZ2Gddde2Z3dS)zMetrics base class.N)AnyDictListOptionalTupleUnion) BaseFileLockTimeout)config)Dataset) ArrowReader) ArrowWriter)DownloadConfig)DownloadManager)Features) DatasetInfo MetricInfo)camelcase_to_snakecase)FileLock) deprecated) get_logger)copyfunc temp_seedcs0eZdZdZfddZddZddZZS) FileFreeLockz-Thread lock until a file **cannot** be lockedcs,t||_tj|jjg|Ri|dSN)rfilelocksuper__init__ lock_file)selfrargskwargs __class__F/var/www/html/lang_env/lib/python3.10/site-packages/datasets/metric.pyr0s "zFileFreeLock.__init__cCsLz |jjdddWnty|jj|j_YdSw|jd|j_dS)Ng{Gz?g{Gz?)timeoutZpoll_intervall)racquirer r_context lock_file_fdreleaser r%r%r&_acquire4s   zFileFreeLock._acquirecCs d|j_dSr)r)r*r,r%r%r&_release?s zFileFreeLock._release)__name__ __module__ __qualname____doc__rr-r. __classcell__r%r%r#r&r-s   rcCsPt|tkr t|dkr|Sdd}d||ddd||dddS) NcSsddd|DS)Nz, css|]}t|VqdSr)repr).0xr%r%r& Jz?summarize_if_long_list..format_chunk..)join)chunkr%r%r& format_chunkIsz,summarize_if_long_list..format_chunk[z, ..., ])typelistlen)objr<r%r%r&summarize_if_long_listEs*rEc@seZdZdZdefddZeddZedefdd Z ede efd d Z edefd d Z edefddZ edefddZedefddZede efddZedefddZede eefddZede eefddZedefddZede efddZd S)!MetricInfoMixinuThis base class exposes some attributes of MetricInfo at the base level of the Metric for easy access. Use the new library 🤗 Evaluate instead: https://huggingface.co/docs/evaluate infocCs ||_dSr _metric_info)r rGr%r%r&r[s zMetricInfoMixin.__init__cCs|jS)zN:class:`datasets.MetricInfo` object containing all the metadata in the metric.rHr,r%r%r&rG^szMetricInfoMixin.inforeturncC|jjSr)rI metric_namer,r%r%r&nameczMetricInfoMixin.namecCrKr)rI experiment_idr,r%r%r&rOgrNzMetricInfoMixin.experiment_idcCrKr)rI descriptionr,r%r%r&rPkrNzMetricInfoMixin.descriptioncCrKr)rIcitationr,r%r%r&rQorNzMetricInfoMixin.citationcCrKr)rIfeaturesr,r%r%r&rRsrNzMetricInfoMixin.featurescCrKr)rIinputs_descriptionr,r%r%r&rSwrNz"MetricInfoMixin.inputs_descriptioncCrKr)rIhomepager,r%r%r&rT{rNzMetricInfoMixin.homepagecCrKr)rIlicenser,r%r%r&rUrNzMetricInfoMixin.licensecCrKr)rI codebase_urlsr,r%r%r&rVrNzMetricInfoMixin.codebase_urlscCrKr)rIreference_urlsr,r%r%r&rWrNzMetricInfoMixin.reference_urlscCrKr)rI streamabler,r%r%r&rXrNzMetricInfoMixin.streamablecCrKr)rIformatr,r%r%r&rYrNzMetricInfoMixin.formatN)r/r0r1r2rrpropertyrGstrrMrrOrPrQrrRrSrTrUrrVrWboolrXrYr%r%r%r&rFOs:  rFc@sxeZdZdZed         d;d eed ed eed ed edeedeedede ee ffddZ ddZ ddZ ddZdMetricuA Metric is the base class and common API for all metrics. Use the new library 🤗 Evaluate instead: https://huggingface.co/docs/evaluate Args: config_name (``str``): This is used to define a hash specific to a metrics computation script and prevents the metric's data to be overridden when the metric loading script is modified. keep_in_memory (:obj:`bool`): keep all predictions and references in memory. Not possible in distributed settings. cache_dir (``str``): Path to a directory in which temporary prediction/references data will be stored. The data directory should be located on a shared file-system in distributed setups. num_process (``int``): specify the total number of nodes in a distributed settings. This is useful to compute metrics in distributed setups (in particular non-additive metrics like F1). process_id (``int``): specify the id of the current process in a distributed setup (between 0 and num_process-1) This is useful to compute metrics in distributed setups (in particular non-additive metrics like F1). seed (:obj:`int`, optional): If specified, this will temporarily set numpy's random seed when :func:`datasets.Metric.compute` is run. experiment_id (``str``): A specific experiment id. This is used if several distributed evaluations share the same file system. This is useful to compute metrics in distributed setups (in particular non-additive metrics like F1). max_concurrent_cache_files (``int``): Max number of concurrent metrics cache files (default 10000). timeout (``Union[int, float]``): Timeout in second for distributed setting synchronization. uOUse the new library 🤗 Evaluate instead: https://huggingface.co/docs/evaluateNFr r'd config_namekeep_in_memory cache_dir num_process process_idseedrOmax_concurrent_cache_filesr'c Ks|pd|_|} t|jj| _|j| _|pd| _t|| t |t r(|dkr,t dt |t r5||kr9t d|rC|dkrCt d||_ ||_ ||_||_tj|pVtj|_||_|durytj^} }} } | dkrs|| n|d|_n||_| |_tt|j ||_ tt|j!||_!tt|j"||_"|j j#j$|j%j&7_$|j!j#j$|j%j&7_$|j"j#j$|j%j&7_$d|_'d|_(d|_)d|_*d|_+d|_,d|_-d|_.d|_/dS) NdefaultZdefault_experimentrz.'process_id' should be a number greater than 0z8'num_process' should be a number greater than process_idr zPUsing 'keep_in_memory' is not possible in distributed setting (num_process > 1).ip)0r`_inforr$r/rLrOrFr isinstanceint ValueErrorrcrdrfraospath expanduserr ZHF_METRICS_CACHE_data_dir_root_build_data_dirdata_dirnprandomZ get_staterer'types MethodTypercompute add_batchadd__func__r2rGrS buf_writerwriterwriter_batch_sizedatacache_file_namerrendez_vous_lock file_paths filelocks)r r`rarbrcrdrerOrfr'r"rG_posr%r%r&rsL      zMetric.__init__cCs|jdurdSt|jS)zReturn the number of examples (predictions or predictions/references pair) currently stored in the metric's cache. Nr)r{rCr,r%r%r&__len__szMetric.__len__c Cs(d|jd|jd|jdt|d S)NzMetric(name: "z ", features: z , usage: """z""", stored examples: ))rMrRrSrCr,r%r%r&__repr__s zMetric.__repr__cCs,|j}tj||j|j}tj|dd|S)aPath of this metric in cache_dir: Will be: self._data_dir_root/self.name/self.config_name/self.hash (if not none)/ If any of these element is missing or if ``with_version=False`` the corresponding subfolders are dropped. T)exist_ok)rorlrmr:rMr`makedirs)r Zbuilder_data_dirr%r%r&rpszMetric._build_data_dirrJcCstj|j|jd|jd|jd}d}t|jD]^}t |d}z|j |dWnIt yt|jdkrDt d|d|jd d||jdkrUt d |jd dt t}tj|j|jd|d|jd|jd}Yqw||fS||fS) zTCreate a new cache file. If the default cache file is used, we generated a new hash.-.arrowN.lockr'r z^Error in _create_cache_file: another metric instance is already using the local cache file at .. Please specify an experiment_id (currently: :) to avoid collision between distributed metric instances.zCannot acquire lock, too many metric instance are operating concurrently on this file system.You should set a larger value of max_concurrent_cache_files when creating the metric (current value is z).)rlrmr:rqrOrcrdrangerfrr(r rkr[uuiduuid4)r r' file_pathriZ file_uuidr%r%r&_create_cache_file s<(    $zMetric._create_cache_filec sjdkrjdurtdjg}n fddtjD}g}t|D]7\}}|dkr4|jq%t|d}z |jj dWnt yVtd |d |d dw||q%||fS) zGet a lock on all the cache files in a distributed setup. We wait for timeout second to let all the distributed node finish their tasks (default is 100 seconds). r Nz|Metric cache file doesn't exist. Please make sure that you call `add` or `add_batch` at least once before calling `compute`.c 2g|]}tjjjdjd|dqS)rrrlrmr:rqrOrcr6rdr,r%r& 8$z/Metric._get_all_cache_files..rrrz#Cannot acquire lock on cached file z for process .) rcr~rkr enumerateappendrrr(r'r )r rrrdrrr%r,r&_get_all_cache_files,s2       zMetric._get_all_cache_filesc srfddtjD}|D](}t|}z |jjdWnty1td|djddw|qdS)Nc r)rz .arrow.lockrrr,r%r&rPrz5Metric._check_all_processes_locks..rExpected to find locked file  from process  but it doesn't exist.) rrcrr(r'r rkrdr+)r Zexpected_lock_file_namesexpected_lock_file_name nofilelockr%r,r&_check_all_processes_locksOs   z!Metric._check_all_processes_locksc Cstj|j|jd|jd}t|}z |j|jdWnt y1t d|d|j ddw| tj|j|jd|jd}t |}z |j|jdWnt ygt d|d|j d dw| dS) Nrz -0.arrow.lockrrrr -rdv.lockzCouldn't acquire lock on r)rlrmr:rqrOrcrr(r'r rkrdr+r)r rrZlock_file_namerr%r%r&_check_rendez_vous_s(     zMetric._check_rendez_vouscCs|jdur |jd|_|jdur|jdkr|j|jr5t|jt|j dd}t |j |_dS|jdkrq|\}}ztdt|j dd}t d i|dd|D|_Wn tyhtddw||_||_dSdS) zClose all the writing process and load/gather the data from all the nodes if main node or all_process is True. NrrR)rmrGcSsg|]}d|iqS)filenamer%)r6fr%r%r&rsz$Metric._finalize..zError in finalize: another metric instance is already using the local cache file. Please specify an experiment_id to avoid collision between distributed metric instances.r%)r{finalizerrdr+rar rqrrRr Z from_bufferrzgetvaluer}rZ read_filesFileNotFoundErrorrkrr)r readerrrr%r%r& _finalizess.     $  zMetric._finalize predictions referencesc  s||d|dur|durfddjD}|nfddjD}|r8td|dtjfd djD}fd dD}td d |Dr_jdi|d_d_ j d kr܈j j j jdfddjD}tjjdi||}Wdn1swYjdurd_` d_ |StttjjD]\} } t d| ` d_ `d_t| | q|SdS)aiCompute the metrics. Usage of positional arguments is not allowed to prevent mistakes. Args: predictions (list/array/tensor, optional): Predictions. references (list/array/tensor, optional): References. **kwargs (optional): Keyword arguments that will be forwarded to the metrics :meth:`_compute` method (see details in the docstring). Return: dict or None - Dictionary with the metrics if this metric is run on the main process (``process_id == 0``). - None if the metric is not run on the main process (``process_id != 0``). Example: ```py >>> from datasets import load_metric >>> metric = load_metric("accuracy") >>> accuracy = metric.compute(predictions=model_prediction, references=labels) ``` rNcsi|] }|vr|dqSrr%r6k all_kwargsr%r& z"Metric.compute..csg|]}|vr|qSr%r%rrr%r&rz"Metric.compute..zMetric inputs are missing: . All required inputs are ci|]}||qSr%r%r6Z input_namerr%r&rcs i|] }|jvr||qSr%rr)r"r r%r&rs css|]}|duVqdSrr%)r6vr%r%r&r8r9z!Metric.compute..r)rAcsi|]}|j|qSr%)r}rr,r%r&rrz Removing r%)rRupdaterkrBanyvaluesrwrr~rrdr}Z set_formatrGrYrre_computerzreversedziprrloggerr{rlremover+) r rrr"Zmissing_kwargsZmissing_inputsZinputsZcompute_kwargsoutputrrr%)rr"r r&rvsJ       zMetric.computec snfdd|D}|rtd|dtj||d|fddjDjjjdur9z jWdStj yt fd d Drt t fd dDd }d dt d|dt |d }n2tjddgkrdjd}dfdd jD}||7}ndjdt|dt|}t|dw)aAdd a batch of predictions and references for the metric's stack. Args: predictions (list/array/tensor, optional): Predictions. references (list/array/tensor, optional): References. Example: ```py >>> from datasets import load_metric >>> metric = load_metric("accuracy") >>> metric.add_batch(predictions=model_prediction, references=labels) ``` cg|] }|jvr|qSr%rrr,r%r&rrz$Metric.add_batch..Bad inputs for metric: rrcrr%r%r6Z intput_namebatchr%r&rrz$Metric.add_batch..Nc3s0|]}t|tttkVqdSr)rCnextiterrr6crr%r&r8s.z#Metric.add_batch..cs(g|]}t|tkr|qSr%)rCr)rcol0r%r&rs(rzMismatch in the number of z (z) and rrr@Metric inputs don't match the expected format. Expected format: , c3(|]}d|dt|VqdSzInput z: NrErrr%r&r8 zPPredictions and/or references don't match the expected format. Expected format: z, Input predictions: z, Input references: )rkrBrRrGZ encode_batchr{ _init_writerZ write_batchpa ArrowInvalidrrrrCsortedr:rE)r rrr" bad_inputsZbad_col error_msgerror_msg_inputsr%)rrr r&rws@  ,  zMetric.add_batch) prediction referencec sfdd|D}|rtd|dtj||d|fddjDjjjdur9z jWdStj ygd jd }d fd d jD}||7}t|dw) aAdd one prediction and reference for the metric's stack. Args: prediction (list/array/tensor, optional): Predictions. reference (list/array/tensor, optional): References. Example: ```py >>> from datasets import load_metric >>> metric = load_metric("accuracy") >>> metric.add(predictions=model_predictions, references=labels) ``` crr%rrr,r%r&rrzMetric.add..rrrcrr%r%rexampler%r&rrzMetric.add..Nrrc3rrrrrr%r&r8&rzMetric.add..) rkrBrRrGZencode_exampler{rwriterrr:)r rrr"rrrr%)rr r&rx s$  z Metric.addc Cs(|jdkr=|jdkr=tj|j|jd|jd}t||_z |jj |dWnt y<t d|d|jddw|j rRt |_t|jj|j|jd |_n%d|_|jdus_|jdurk|\}}||_||_t|jj|j|jd |_|jdkr|jdkr||jdS|dSdS) Nr rrrrzXError in _init_writer: another metric instance is already using the local cache file at rr)rRstreamr|)rRrmr|)rcrdrlrmr:rqrOrrr( TimeoutErrorrkrarZBufferOutputStreamrzrrGrRr|r{r~rrrr+r)r r'rr~rr%r%r&r,sD           zMetric._init_writercCt)aConstruct the MetricInfo object. See `MetricInfo` for details. Warning: This function is only called once and the result is cached for all following .info() calls. Returns: info: (MetricInfo) The metrics information NotImplementedErrorr,r%r%r&rhS z Metric._infodownload_config dl_managercCsN|dur |durt}tj|jd|_d|_t|j||jd}| |dS)aDownloads and prepares dataset for reading. Args: download_config (:class:`DownloadConfig`, optional): Specific download configuration parameters. dl_manager (:class:`DownloadManager`, optional): Specific download manager to use. NZ downloadsF)Z dataset_namerrq) rrlrmr:rqrbZforce_downloadrrM_download_and_prepare)r rrr%r%r&download_and_prepare^s  zMetric.download_and_preparecCsdS)aYDownloads and prepares resources for the metric. This is the internal implementation to overwrite called when user calls `download_and_prepare`. It should download all required resources for the metric. Args: dl_manager (:class:`DownloadManager`): `DownloadManager` used to download and cache data. Nr%)r rr%r%r&rurzMetric._download_and_preparecKr)zEThis method defines the common API for all the metrics in the libraryr)r rrr"r%r%r&rszMetric._computecCs`t|dr|jdur|jt|dr|jdur|jt|dr%|`t|dr.|`dSdS)Nrrr{r})hasattrrr+rr{r}r,r%r%r&__del__s    zMetric.__del__) NFNr rNNr^r_)r )NN)&r/r0r1r2rrr[r\rjrfloatrrrrprrrrrrrrdictrvrwrxrrrhrrrrrrrrr%r%r%r&r]sl  D !#"G/ !'   r])4r2rlrtrtypingrrrrrrnumpyrrZpyarrowrrrr rr Z arrow_datasetr Z arrow_readerr Z arrow_writerrZdownload.download_configrZdownload.download_managerrrRrrGrrZnamingrZutils._filelockrZutils.deprecation_utilsrZ utils.loggingrZutils.py_utilsrrr/rrrErFr]r%r%r%r&s4             E