o +ifm@sBdZddlZddlZddlZddlmZmZmZmZm Z m Z ddl Z ddl ZddlmZmZddlmZddlmZddlmZdd lmZdd lmZdd lmZdd lmZdd l m!Z!m"Z"ddl#m$Z$ddl%m&Z&ddl'm(Z(ddl)m*Z*ddl+m,Z,m-Z-e*e.Z/GdddeZ0ddZ1GdddZ2Gddde2Z3dS)zMetrics base class.N)AnyDictListOptionalTupleUnion) BaseFileLockTimeout)config)Dataset) ArrowReader) ArrowWriter)DownloadConfig)DownloadManager)Features) DatasetInfo MetricInfo)camelcase_to_snakecase)FileLock) deprecated) get_logger)copyfunc temp_seedcs0eZdZdZfddZddZddZZS) FileFreeLockz-Thread lock until a file **cannot** be lockedcs,t||_tj|jjg|Ri|dSN)rfilelocksuper__init__ lock_file)selfrargskwargs __class__H/var/www/html/corbot_env/lib/python3.10/site-packages/datasets/metric.pyr0s "zFileFreeLock.__init__cCsLz |jjdddWnty|jj|j_YdSw|jd|j_dS)Ng{Gz?g{Gz?)timeoutpoll_intervall)racquirer r_context lock_file_fdreleaser r%r%r&_acquire4s   zFileFreeLock._acquirecCs d|j_dSr)r*r+r-r%r%r&_release?s zFileFreeLock._release)__name__ __module__ __qualname____doc__rr.r/ __classcell__r%r%r#r&r-s   rcCsPt|tkr t|dkr|Sdd}d||ddd||dddS) NcSsddd|DS)Nz, css|]}t|VqdSr)repr).0xr%r%r& Jz?summarize_if_long_list..format_chunk..)join)chunkr%r%r& format_chunkIsz,summarize_if_long_list..format_chunk[z, ..., ])typelistlen)objr=r%r%r&summarize_if_long_listEs*rFc@seZdZdZdefddZeddZedefdd Z ede efd d Z edefd d Z edefddZ edefddZedefddZede efddZedefddZede eefddZede eefddZedefddZede efddZd S)!MetricInfoMixinuThis base class exposes some attributes of MetricInfo at the base level of the Metric for easy access. Use the new library 🤗 Evaluate instead: https://huggingface.co/docs/evaluate infocCs ||_dSr _metric_info)r rHr%r%r&r[s zMetricInfoMixin.__init__cCs|jS)zN:class:`datasets.MetricInfo` object containing all the metadata in the metric.rIr-r%r%r&rH^szMetricInfoMixin.inforeturncC|jjSr)rJ metric_namer-r%r%r&nameczMetricInfoMixin.namecCrLr)rJ experiment_idr-r%r%r&rPgrOzMetricInfoMixin.experiment_idcCrLr)rJ descriptionr-r%r%r&rQkrOzMetricInfoMixin.descriptioncCrLr)rJcitationr-r%r%r&rRorOzMetricInfoMixin.citationcCrLr)rJfeaturesr-r%r%r&rSsrOzMetricInfoMixin.featurescCrLr)rJinputs_descriptionr-r%r%r&rTwrOz"MetricInfoMixin.inputs_descriptioncCrLr)rJhomepager-r%r%r&rU{rOzMetricInfoMixin.homepagecCrLr)rJlicenser-r%r%r&rVrOzMetricInfoMixin.licensecCrLr)rJ codebase_urlsr-r%r%r&rWrOzMetricInfoMixin.codebase_urlscCrLr)rJreference_urlsr-r%r%r&rXrOzMetricInfoMixin.reference_urlscCrLr)rJ streamabler-r%r%r&rYrOzMetricInfoMixin.streamablecCrLr)rJformatr-r%r%r&rZrOzMetricInfoMixin.formatN)r0r1r2r3rrpropertyrHstrrNrrPrQrRrrSrTrUrVrrWrXboolrYrZr%r%r%r&rGOs:  rGc@sxeZdZdZed         d;d eed ed eed ed edeedeedede ee ffddZ ddZ ddZ ddZdMetricuA Metric is the base class and common API for all metrics. Use the new library 🤗 Evaluate instead: https://huggingface.co/docs/evaluate Args: config_name (``str``): This is used to define a hash specific to a metrics computation script and prevents the metric's data to be overridden when the metric loading script is modified. keep_in_memory (:obj:`bool`): keep all predictions and references in memory. Not possible in distributed settings. cache_dir (``str``): Path to a directory in which temporary prediction/references data will be stored. The data directory should be located on a shared file-system in distributed setups. num_process (``int``): specify the total number of nodes in a distributed settings. This is useful to compute metrics in distributed setups (in particular non-additive metrics like F1). process_id (``int``): specify the id of the current process in a distributed setup (between 0 and num_process-1) This is useful to compute metrics in distributed setups (in particular non-additive metrics like F1). seed (:obj:`int`, optional): If specified, this will temporarily set numpy's random seed when :func:`datasets.Metric.compute` is run. experiment_id (``str``): A specific experiment id. This is used if several distributed evaluations share the same file system. This is useful to compute metrics in distributed setups (in particular non-additive metrics like F1). max_concurrent_cache_files (``int``): Max number of concurrent metrics cache files (default 10000). timeout (``Union[int, float]``): Timeout in second for distributed setting synchronization. uOUse the new library 🤗 Evaluate instead: https://huggingface.co/docs/evaluateNFr r'd config_namekeep_in_memory cache_dir num_process process_idseedrPmax_concurrent_cache_filesr'c Ks|pd|_|} t|jj| _|j| _|pd| _t|| t |t r(|dkr,t dt |t r5||kr9t d|rC|dkrCt d||_ ||_ ||_||_tj|pVtj|_||_|durytj^} }} } | dkrs|| n|d|_n||_| |_tt|j ||_ tt|j!||_!tt|j"||_"|j j#j$|j%j&7_$|j!j#j$|j%j&7_$|j"j#j$|j%j&7_$d|_'d|_(d|_)d|_*d|_+d|_,d|_-d|_.d|_/dS) Ndefaultdefault_experimentrz.'process_id' should be a number greater than 0z8'num_process' should be a number greater than process_idr zPUsing 'keep_in_memory' is not possible in distributed setting (num_process > 1).ip)0ra_inforr$r0rMrPrGr isinstanceint ValueErrorrdrergrbospath expanduserr HF_METRICS_CACHE_data_dir_root_build_data_dirdata_dirnprandom get_staterfr'types MethodTypercompute add_batchadd__func__r3rHrT buf_writerwriterwriter_batch_sizedatacache_file_namerrendez_vous_lock file_paths filelocks)r rarbrcrdrerfrPrgr'r"rH_posr%r%r&rsL      zMetric.__init__cCs|jdurdSt|jS)zReturn the number of examples (predictions or predictions/references pair) currently stored in the metric's cache. Nr)rrDr-r%r%r&__len__szMetric.__len__c Cs(d|jd|jd|jdt|d S)NzMetric(name: "z ", features: z , usage: """z""", stored examples: ))rNrSrTrDr-r%r%r&__repr__s zMetric.__repr__cCs,|j}tj||j|j}tj|dd|S)aPath of this metric in cache_dir: Will be: self._data_dir_root/self.name/self.config_name/self.hash (if not none)/ If any of these element is missing or if ``with_version=False`` the corresponding subfolders are dropped. T)exist_ok)rrrnror;rNramakedirs)r builder_data_dirr%r%r&rsszMetric._build_data_dirrKcCstj|j|jd|jd|jd}d}t|jD]^}t |d}z|j |dWnIt yt|jdkrDt d|d|jd d||jdkrUt d |jd dt t}tj|j|jd|d|jd|jd}Yqw||fS||fS) zTCreate a new cache file. If the default cache file is used, we generated a new hash.-.arrowN.lockr'r z^Error in _create_cache_file: another metric instance is already using the local cache file at .. Please specify an experiment_id (currently: :) to avoid collision between distributed metric instances.zCannot acquire lock, too many metric instance are operating concurrently on this file system.You should set a larger value of max_concurrent_cache_files when creating the metric (current value is z).)rnror;rtrPrdrerangergrr)r rmr\uuiduuid4)r r' file_pathri file_uuidr%r%r&_create_cache_file s<(    $zMetric._create_cache_filec sjdkrjdurtdjg}n fddtjD}g}t|D]7\}}|dkr4|jq%t|d}z |jj dWnt yVtd |d |d dw||q%||fS) zGet a lock on all the cache files in a distributed setup. We wait for timeout second to let all the distributed node finish their tasks (default is 100 seconds). r Nz|Metric cache file doesn't exist. Please make sure that you call `add` or `add_batch` at least once before calling `compute`.c 2g|]}tjjjdjd|dqS)rrrnror;rtrPrdr7rer-r%r& 8$z/Metric._get_all_cache_files..rrrz#Cannot acquire lock on cached file z for process .) rdrrmr enumerateappendrrr)r'r )r rrrerrr%r-r&_get_all_cache_files,s2       zMetric._get_all_cache_filesc srfddtjD}|D](}t|}z |jjdWnty1td|djddw|qdS)Nc r)rz .arrow.lockrrr-r%r&rPrz5Metric._check_all_processes_locks..rExpected to find locked file  from process  but it doesn't exist.) rrdrr)r'r rmrer,)r expected_lock_file_namesexpected_lock_file_name nofilelockr%r-r&_check_all_processes_locksOs   z!Metric._check_all_processes_locksc Cstj|j|jd|jd}t|}z |j|jdWnt y1t d|d|j ddw| tj|j|jd|jd}t |}z |j|jdWnt ygt d|d|j d dw| dS) Nrz -0.arrow.lockrrrr -rdv.lockzCouldn't acquire lock on r)rnror;rtrPrdrr)r'r rmrer,r)r rrlock_file_namerr%r%r&_check_rendez_vous_s(     zMetric._check_rendez_vouscCs|jdur |jd|_|jdur|jdkr|j|jr5t|jt|j dd}t |j |_dS|jdkrq|\}}ztdt|j dd}t d i|dd|D|_Wn tyhtddw||_||_dSdS) zClose all the writing process and load/gather the data from all the nodes if main node or all_process is True. NrrS)rorHcSsg|]}d|iqS)filenamer%)r7fr%r%r&rsz$Metric._finalize..zError in finalize: another metric instance is already using the local cache file. Please specify an experiment_id to avoid collision between distributed metric instances.r%)rfinalizerrer,rbr rtrrSr from_bufferr~getvaluerr read_filesFileNotFoundErrorrmrr)r readerrrr%r%r& _finalizess.     $  zMetric._finalize predictions referencesc  s||d|dur|durfddjD}|nfddjD}|r8td|dtjfd djD}fd dD}td d |Dr_jdi|d_d_ j d kr܈j j j jdfddjD}tjjdi||}Wdn1swYjdurd_` d_ |StttjjD]\} } t d| ` d_ `d_t| | q|SdS)aiCompute the metrics. Usage of positional arguments is not allowed to prevent mistakes. Args: predictions (list/array/tensor, optional): Predictions. references (list/array/tensor, optional): References. **kwargs (optional): Keyword arguments that will be forwarded to the metrics :meth:`_compute` method (see details in the docstring). Return: dict or None - Dictionary with the metrics if this metric is run on the main process (``process_id == 0``). - None if the metric is not run on the main process (``process_id != 0``). Example: ```py >>> from datasets import load_metric >>> metric = load_metric("accuracy") >>> accuracy = metric.compute(predictions=model_prediction, references=labels) ``` rNcsi|] }|vr|dqSrr%r7k all_kwargsr%r& z"Metric.compute..csg|]}|vr|qSr%r%rrr%r&rz"Metric.compute..zMetric inputs are missing: . All required inputs are ci|]}||qSr%r%r7 input_namerr%r&rcs i|] }|jvr||qSr%rr)r"r r%r&rs css|]}|duVqdSrr%)r7vr%r%r&r9r:z!Metric.compute..r)rBcsi|]}|j|qSr%)rrr-r%r&rrz Removing r%)rSupdatermrCanyvaluesr{rrrrer set_formatrHrZrrf_computer~reversedziprrloggerrrnremover,) r rrr"missing_kwargsmissing_inputsinputscompute_kwargsoutputrrr%)rr"r r&rzsJ       zMetric.computec snfdd|D}|rtd|dtj||d|fddjDjjjdur9z jWdStj yt fd d Drt t fd dDd }d dt d|dt |d }n2tjddgkrdjd}dfdd jD}||7}ndjdt|dt|}t|dw)aAdd a batch of predictions and references for the metric's stack. Args: predictions (list/array/tensor, optional): Predictions. references (list/array/tensor, optional): References. Example: ```py >>> from datasets import load_metric >>> metric = load_metric("accuracy") >>> metric.add_batch(predictions=model_prediction, references=labels) ``` cg|] }|jvr|qSr%rrr-r%r&rrz$Metric.add_batch..Bad inputs for metric: rrcrr%r%r7 intput_namebatchr%r&rrz$Metric.add_batch..Nc3s0|]}t|tttkVqdSr)rDnextiterrr7crr%r&r9s.z#Metric.add_batch..cs(g|]}t|tkr|qSr%)rDr)rcol0r%r&rs(rzMismatch in the number of z (z) and rrr@Metric inputs don't match the expected format. Expected format: , c3(|]}d|dt|VqdSzInput z: NrFrrr%r&r9 zPPredictions and/or references don't match the expected format. Expected format: z, Input predictions: z, Input references: )rmrCrSrH encode_batchr _init_writer write_batchpa ArrowInvalidrrrrDsortedr;rF)r rrr" bad_inputsbad_col error_msgerror_msg_inputsr%)rrr r&r{s@  ,  zMetric.add_batch) prediction referencec sfdd|D}|rtd|dtj||d|fddjDjjjdur9z jWdStj ygd jd }d fd d jD}||7}t|dw) aAdd one prediction and reference for the metric's stack. Args: prediction (list/array/tensor, optional): Predictions. reference (list/array/tensor, optional): References. Example: ```py >>> from datasets import load_metric >>> metric = load_metric("accuracy") >>> metric.add(predictions=model_predictions, references=labels) ``` crr%rrr-r%r&rrzMetric.add..rrrcrr%r%rexampler%r&rrzMetric.add..Nrrc3rrrrrr%r&r9&rzMetric.add..) rmrCrSrHencode_examplerrwriterrr;)r rrr"rrrr%)rr r&r| s$  z Metric.addc Cs(|jdkr=|jdkr=tj|j|jd|jd}t||_z |jj |dWnt y<t d|d|jddw|j rRt |_t|jj|j|jd |_n%d|_|jdus_|jdurk|\}}||_||_t|jj|j|jd |_|jdkr|jdkr||jdS|dSdS) Nr rrrrzXError in _init_writer: another metric instance is already using the local cache file at rr)rSstreamr)rSror)rdrernror;rtrPrrr) TimeoutErrorrmrbrBufferOutputStreamr~rrHrSrrrrrrr,r)r r'rrrr%r%r&r,sD           zMetric._init_writercCt)aConstruct the MetricInfo object. See `MetricInfo` for details. Warning: This function is only called once and the result is cached for all following .info() calls. Returns: info: (MetricInfo) The metrics information NotImplementedErrorr-r%r%r&rjS z Metric._infodownload_config dl_managercCsN|dur |durt}tj|jd|_d|_t|j||jd}| |dS)aDownloads and prepares dataset for reading. Args: download_config (:class:`DownloadConfig`, optional): Specific download configuration parameters. dl_manager (:class:`DownloadManager`, optional): Specific download manager to use. N downloadsF) dataset_namerrt) rrnror;rtrcforce_downloadrrN_download_and_prepare)r rrr%r%r&download_and_prepare^s  zMetric.download_and_preparecCsdS)aYDownloads and prepares resources for the metric. This is the internal implementation to overwrite called when user calls `download_and_prepare`. It should download all required resources for the metric. Args: dl_manager (:class:`DownloadManager`): `DownloadManager` used to download and cache data. Nr%)r rr%r%r&rurzMetric._download_and_preparecKr)zEThis method defines the common API for all the metrics in the libraryr)r rrr"r%r%r&rszMetric._computecCs`t|dr|jdur|jt|dr|jdur|jt|dr%|`t|dr.|`dSdS)Nrrrr)hasattrrr,rrrr-r%r%r&__del__s    zMetric.__del__) NFNr rNNr_r`)r )NN)&r0r1r2r3rrr\r]rlrfloatrrrrsrrrrrrrrdictrzr{r|rrrjrrrrrrrrr%r%r%r&r^sl  D !#"G/ !'   r^)4r3rnrxrtypingrrrrrrnumpyrupyarrowrrrr rr arrow_datasetr arrow_readerr arrow_writerrdownload.download_configrdownload.download_managerrrSrrHrrnamingrutils._filelockrutils.deprecation_utilsr utils.loggingrutils.py_utilsrrr0rrrFrGr^r%r%r%r&s4             E