o +if=)@sddlZddlZddlZddlZddlZddlmZddlmZddl m Z ddl m Z m Z ddlZddlmZmZddlmZddlmZmZmZmZmZmZmZmZmZmZdd lm Z  dd e!fd d Z"       dd e!fddZ#dS)N)contextmanager)partial)Path)urljoinurlparse)DownloadConfigconfig)ExtractManager) !_raise_if_offline_mode_is_enabledftp_getftp_head"get_authentication_headers_for_urlhash_url_to_filenamehttp_get http_head is_local_path is_remote_urllogger)FileLockreturncKs|dur td i|}|jptj}t|trt|}t|tr#t|}t|rBt|||j |j |j |j |j |j|j|j|j|jd }ntj|rK|}nt|rWtd|dtd|d|dure|S|jrtt|jdj||jd}|S) a Given something that might be a URL (or might be a local path), determine which. If it's a URL, download the file and cache it, and return the path to the cached file. If it's already a local path, make sure the file exists and then return the path. Return: Local path (string) Raises: FileNotFoundError: in case of non-recoverable file (non-existent or no cache on disk) ConnectionError: in case of unreachable url and no cache on disk ValueError: if it couldn't parse the url or filename correctly requests.exceptions.ConnectionError: in case of internet connection issue N) cache_dirforce_downloadproxiesresume_download user_agentlocal_files_onlyuse_etag max_retriesuse_auth_tokenignore_url_params download_descz Local file z doesn't existzunable to parse z as a URL or as a local path)r) force_extract)rrrDOWNLOADED_DATASETS_PATH isinstancerstrrget_from_cacherrrrrrrrrr ospathexistsrFileNotFoundError ValueErrorextract_compressed_filer extractr!)url_or_filenamedownload_configdownload_kwargsr output_pathr"r"M/var/www/html/corbot_env/lib/python3.10/site-packages/datasets/utils/cache.py cached_pathsD     r3FdTc ! s|durtj}t|trt|}tj|dd| r"t|t|j } n|} d}d}d}d}d}t | dd}tj ||}tj |rG|sG|sG|St || d}|durU||d<|s|drat|}zt|d||| |d }|jd kr|ry|jd nd}|jD]\}}|d rd |vr|d|7}|j}qd}d |vrd|vr|d7}nN|jdkrd|vs|jdkrd |vs|jdkrtd|std|js|jdkrd|vrd}td|n|jdkrtj|vr| durtd|dWnttjj fy}z|}WYd}~nd}~ww|sltj |r|s|S|r)t!d|d|dur;|jdkr;t!d|t"d ||durTtd!|d"t#|d#|duretd!|d$|jd#td!|t | |}tj ||}tj |r|s|S|d%}t$||r|d&t%fd'd(}|}tj rt&j'}n d)}n t(t)j*|dd*}d)}|+}t|d+|j+|drt,||n t-||||||| | d,Wdn 1swYtd-|d.|t./|j+|td/|||d0}|d1}t0|d2d3d4} t12|| Wdn1s,wYWd|SWd|S1sEwY|S)5a Given a URL, look for the corresponding file in the local cache. If it's not there, download it. Then return the path to the cached file. Return: Local path (string) Raises: FileNotFoundError: in case of non-recoverable file (non-existent or no cache on disk) ConnectionError: in case of unreachable url and no cache on disk NT)exist_okF)etag)rz user-agentzftp://)allow_redirectsrtimeoutrheadersETagdownload_warningzdrive.google.comz &confirm=zconfirm=z &confirm=tizfirebasestorage.googleapis.comiiz7^https?://github.com/.*?/.*?/releases/download/.*?/.*?$z#^https://.*?s3.*?amazonaws.com/.*?$zndownloader.figstatic.comz"Couldn't get ETag version for url izUnauthorized for URL z^. Please use the parameter `use_auth_token=True` after logging in with `huggingface-cli login`z6Cannot find the requested files in the cached path at zi and outgoing traffic has been disabled. To enable file online look-ups, set 'local_files_only' to False.izCouldn't find file at zTried to reach zCouldn't reach z ()z (error z.lockz .incompletec3s8td }|VWddS1swYdS)Nza+b)open)fincomplete_pathr"r2_resumable_file_managers "z/get_from_cache.._resumable_file_managerr)dirdeletezB not found in cache or force_download set to True, downloading to )r resume_sizer9cookiesrdesczstoring z in cache at zcreating metadata file for )urlr6z.jsonwzutf-8)encoding)3rHF_DATASETS_CACHEr$rr%r'makedirsrrr(rjoinr)r startswithr r status_coder9getrFitemsrematchrHrinfo HF_ENDPOINTConnectionErrorOSErrorrequests exceptionsTimeoutr*r reprrrstatst_sizertempfileNamedTemporaryFilenamer rshutilmover>jsondump)!rHrrr etag_timeoutrrrrrrrr cached_url connectedresponserFr6 head_errorfilename cache_pathr9kve lock_pathrBtemp_file_managerrE temp_filemeta meta_path meta_filer"r@r2r&bs                    / //r&)N) NFNr4FNFTrNFN)$rcr'rRrar^ contextlibr functoolsrpathlibr urllib.parserrrXdatasetsrrdatasets.utils.extractr datasets.utils.file_utilsr r r r rrrrrrdatasets.utils.filelockrr%r3r&r"r"r"r2sB    0  F