o "ifp @sUddlmZddlZddlZddlZddlZddlZddlZddlZddl m Z m Z ddl m Z mZmZmZmZmZmZmZe rPddlZddlmZedZnedZedZedZeeegefZGd d d ZGd d d eZ Gd ddeZ!GdddeZ"GdddeZ#GdddeZ$GdddeZ%GdddeZ&GdddeeefZ'GdddeZ(deiZ)de*d<d(d)d&d'Z+ee e$e!e#e"e%e&e(f D]Z,e+e,qdS)*) annotationsN)FutureThreadPoolExecutor) TYPE_CHECKINGAnyCallableClassVarGeneric NamedTuple OrderedDictTypeVar) ParamSpecPTfsspecc@s2eZdZUdZdZded<dd d ZdddZdS) BaseCacheagPass-though cache: doesn't keep anything, calls every time Acts as base class for other cachers Parameters ---------- blocksize: int How far to read ahead in numbers of bytes fetcher: func Function of the form f(start, end) which gets bytes from remote as specified size: int How big this file is none ClassVar[str]name blocksizeintfetcherFetchersizereturnNonecCs||_||_||_dSN)rrrselfrrrrG/var/www/html/corbot_env/lib/python3.10/site-packages/fsspec/caching.py__init__9s zBaseCache.__init__start int | NonestopbytescCs<|durd}|dur |j}||jks||krdS|||SNr)rrrr"r$rrr _fetch>s zBaseCache._fetchNrrrrrrrrr"r#r$r#rr%)__name__ __module__ __qualname____doc__r__annotations__r!r)rrrr r's   rcsTeZdZdZdZ  d d!fdd Zd"ddZd#ddZd$ddZd%ddZ Z S)& MMapCachezmemory-mapped sparse file cache Opens temporary file, which is filled blocks-wise when data is requested. Ensure there is enough disc space in the temporary location. This cache method might only work on posix mmapNrrrrrlocation str | Noneblocksset[int] | Nonerrcs8t||||durtn||_||_||_dSr)superr!setr5r3 _makefilecache)rrrrr3r5 __class__rr r!SszMMapCache.__init__mmap.mmap | bytearraycCsddl}ddl}|jdkrtS|jdustj|jsB|jdur*|}t |_ nt |jd}| |jd| d|nt |jd}|||jS)Nrzwb+1zr+b)r2tempfiler bytearrayr3ospathexists TemporaryFiler8r5openseekwriteflushfileno)rr2r@fdrrr r9`s       zMMapCache._makefiler"r#endr%c std|d||durd}|durj}|jks!||kr#dS|j}|j}fddt||dD}|rq|d}|j}t|jj}td|d |d|||j||<j ||s=j||S) NzMMap cache fetching -rr'csg|] }|jvr|qSr)r5).0irrr sz$MMapCache._fetch..r>zMMap get block #z () loggerdebugrrrangepopminrr:r5add) rr"rL start_block end_blockneedrOsstartsendrrPr r)vs&      zMMapCache._fetchdict[str, Any]cCs|j}|d=|S)Nr:)__dict__copyrstaterrr __getstate__s zMMapCache.__getstate__racCs|j|||_dSr)r^updater9r:r`rrr __setstate__s zMMapCache.__setstate__)NN) rrrrrrr3r4r5r6rr)rr=r"r#rLr#rr%rr]rar]rr) r,r-r.r/rr!r9r)rbrd __classcell__rrr;r r1Hs  r1c0eZdZdZdZdfd d ZdddZZS)ReadAheadCachea!Cache which reads only when we get beyond a block of data This is a much simpler version of BytesCache, and does not attempt to fill holes in the cache or keep fragments alive. It is best suited to many small reads in a sequential order (e.g., reading lines from a file). readaheadrrrrrrrcs&t|||d|_d|_d|_dS)Nr'r)r7r!r:r"rLrr;rr r!s zReadAheadCache.__init__r"r#rLr%cCs|durd}|dus||jkr|j}||jks||krdS||}||jkr8||jkr8|j||j||jS|j|krD|jkrZnn|j||jd}|t|8}|j}nd}t|j||j}||||_||_|jt|j|_||jd|Sr&)rr"rLr:lenrVrr)rr"rLlpartrrr r)s& zReadAheadCache._fetchr*rer,r-r.r/rr!r)rhrrr;r rjs rjcri)FirstChunkCachezCaches the first block of a file only This may be useful for file types where the metadata is stored in the header, but is randomly accessed. firstrrrrrrrcst|||d|_dSr)r7r!r:rr;rr r!s zFirstChunkCache.__init__r"r#rLr%cCs|pd}|p|j}||jkrK|jdur4||jkr,|d|}|d|j|_||dS|d|j|_|j||}||jkrI|||j|7}|S|||SNr)rrr:r)rr"rLdatarnrrr r)s        zFirstChunkCache._fetchr*rerorrr;r rps rpcsreZdZdZdZ d&d'fd d Zd(ddZddZd)ddZd*ddZ d+ddZ d,fd d! Z d-d$d%Z Z S). BlockCachea Cache holding memory as a set of blocks. Requests are only ever made ``blocksize`` at a time, and are stored in an LRU cache. The least recently accessed block is discarded when more than ``maxblocks`` are stored. Parameters ---------- blocksize : int The number of bytes to store in each block. Requests are only ever made for ``blocksize``, so this should balance the overhead of making a request against the granularity of the blocks. fetcher : Callable size : int The total size of the file being cached. maxblocks : int The maximum number of blocks to cache for. The maximum memory use for this cache is then ``blocksize * maxblocks``. blockcache rrrrr maxblocksrrcs<t|||t|||_||_t||j|_ dSr) r7r!mathceilnblocksrw functools lru_cache _fetch_block_fetch_block_cachedrrrrrwr;rr r!szBlockCache.__init__strcCd|jd|jd|jdS)NzrrrzrPrrr __repr__ zBlockCache.__repr__cC |jSz The statistics on the block cache. Returns ------- NamedTuple Returned directly from the LRU Cache used internally. r~ cache_inforPrrr r  zBlockCache.cache_infor]cCs|j}|d=|S)Nr~r^r`rrr rbszBlockCache.__getstate__racCs&|j|t|d|j|_dS)Nrw)r^rcr{r|r}r~r`rrr rds   zBlockCache.__setstate__r"r#rLr%cCst|durd}|dur |j}||jks||krdS||j}||j}t||dD]}||q)|j||||dS)Nrr'r>start_block_numberend_block_number)rrrTr~ _read_cache)rr"rLrr block_numberrrr r)s    zBlockCache._fetchrcsT||jkrtd|d|jd||j}||j}td|t||}|S)= Fetch the block of data for `block_number`. 'block_number=(' is greater than the number of blocks ()zBlockCache fetching block %drz ValueErrorrrRinfor7r))rrr"rLblock_contentsr;rr r}6s    zBlockCache._fetch_blockrrc C||j}||j}||kr||}|||S|||dg}|t|jt|d||||d|d|Sz Read from our block cache. Parameters ---------- start, end : int The start and end byte positions. start_block_number, end_block_number : int The start and end block numbers. Nr>r'rr~extendmaprTappendjoin rr"rLrr start_posend_posblockoutrrr rF     zBlockCache._read_cacherv rrrrrrrwrrrrrrfrgre)rrrr% r"rrLrrrrrrr%)r,r-r.r/rr!rrrbrdr)r}rrhrrr;r rts   rtcsHeZdZUdZdZded< ddfdd ZdddZdddZZ S) BytesCacheaKCache which holds data in a in-memory bytes object Implements read-ahead by the block size, for semi-random reads progressing through the file. Parameters ---------- trim: bool As we read more data, whether to discard the start of the buffer when we are more than a blocksize ahead of it. r%rrTrrrrrtrimboolrrcs,t|||d|_d|_d|_||_dS)Nr')r7r!r:r"rLr)rrrrrr;rr r!}s  zBytesCache.__init__r"r#rLcCsL|durd}|dur |j}||jks||krdS|jdur<||jkr<|jdur<||jkr<||j}|j||||S|jrIt|j||j}n|}||ksT||jkrVdS|jdus`||jkru|jdusj||jkru||||_||_nm|jdus|J|jdusJ||jkr|jdus|j||jkr||||_||_nB|||j}||_||j|_n1|jdur||jkr|j|jkrn ||j|jkr||||_||_n ||j|}|j||_|jt|j|_||j}|j||||}|jr$|j|j|jd}|dkr$|j|j|7_|j|j|d|_|S)Nrr'r>) rr"rLr:rrVrrlr)rr"rLoffsetbendnewrnumrrr r)s^          zBytesCache._fetchcCs t|jSr)rlr:rPrrr __len__s zBytesCache.__len__)T) rrrrrrrrrrre)rr) r,r-r.r/rr0r!r)rrhrrr;r rns   ArcsDeZdZUdZdZded<    ddfdd ZdddZZS)AllBytesz!Cache entire contents of the fileallrrNrr#rFetcher | Nonerrs bytes | Nonerrcs0t||||dur|d|j}||_dSrr)r7r!rrrs)rrrrrsr;rr r!s zAllBytes.__init__r"r$r%cCs|j||Sr)rsr(rrr r)szAllBytes._fetch)NNNN) rr#rrrr#rsrrrr+ r,r-r.r/rr0r!r)rhrrr;r rs   rcsDeZdZUdZdZded<idfdfdd Zdfdd ZZS)KnownPartsOfAFilea Cache holding known file parts. Parameters ---------- blocksize: int How far to read ahead in numbers of bytes fetcher: func Function of the form f(start, end) which gets bytes from remote as specified size: int How big this file is data: dict A dictionary mapping explicit `(start, stop)` file-offset tuples with known bytes. strict: bool, default True Whether to fetch reads that go beyond a known byte-range boundary. If `False`, any read that ends outside a known part will be zero padded. Note that zero padding will not be used for reads that begin outside a known byte-range. partsrrTrrrrrrsdict[tuple[int, int], bytes]strictr_rc st|||||_|rdt|}|dg}||dg} |ddD]3\} } |d\} } | | krH| | f|d<| d|| | f7<q&|| | f| || | fq&tt|| |_ dS||_ dS)Nrr>) r7r!rsortedkeysrUrdictziprs)rrrrrsrr old_offsetsoffsetsr5r"r$start0stop0r;rr r!s      zKnownPartsOfAFile.__init__r"r#r$rr%cs |durd}|dur |j}d}|jD]C\\}}}||kr$|krWnq||}|||||}|jrC||krA|krSnn|d||t|7}|S|}nq|jdurgtd||fdtd||fdt d|d||t ||S) Nrr'z&Read is outside the known file parts: z. z%. IO/caching performance may be poor!z!KnownPartsOfAFile cache fetching rM) rrsitemsrrlrrwarningswarnrRrSr7r))rr"r$rloc0loc1rsoffr;rr r)s, zKnownPartsOfAFile._fetch) rrrrrrrsrrrrrr+rrrr;r rs  rc@sTeZdZdZGdddeZdd d d Zd!ddZd"ddZd#ddZ d$ddZ dS)% UpdatableLRUzg Custom implementation of LRU cache that allows updating keys Used by BackgroudBlockCache c@s.eZdZUded<ded<ded<ded<dS)UpdatableLRU.CacheInforhitsmissesmaxsizecurrsizeN)r,r-r.r0rrrr CacheInfoIs  rfuncCallable[P, T]max_sizerrrcCs0t|_||_||_d|_d|_t|_ dSrr) collectionsr _cache_func _max_size_hits_misses threadingLock_lock)rrrrrr r!Os zUpdatableLRU.__init__argsP.argskwargsP.kwargsrcOs|r td||j&||jvr-|j||jd7_|j|WdSWdn1s7wY|j|i|}|j,||j|<|jd7_t|j|j krk|jj ddWd|SWd|S1svwY|S)Nz Got unexpected keyword argument r>Flast) TypeErrorrrr move_to_endrrrrlrpopitem)rrrresultrrr __call__Ws.     zUpdatableLRU.__call__rrcGs4|j ||jvWdS1swYdSr)rr)rrrrr is_key_cachedjs$zUpdatableLRU.is_key_cachedrcGsd|j%||j|<t|j|jkr |jjddWddSWddS1s+wYdS)NFr)rrrlrr)rrrrrr add_keyns "zUpdatableLRU.add_keyrcCsH|j|j|jt|j|j|jdWdS1swYdS)N)rrrr)rrrrlrrrrPrrr rts$zUpdatableLRU.cache_infoN)r)rrrrrr)rrrrrr)rrrr)rrrrrrrr) r,r-r.r/r rr!rrrrrrrr rBs    rcseZdZUdZdZded< d*d+fdd Zd,ddZd-ddZd.ddZ d/ddZ d0dd Z d1d2fd$d% Z d3d(d)Z ZS)4BackgroundBlockCachea Cache holding memory as a set of blocks with pre-loading of the next block in the background. Requests are only ever made ``blocksize`` at a time, and are stored in an LRU cache. The least recently accessed block is discarded when more than ``maxblocks`` are stored. If the next block is not in cache, it is loaded in a separate thread in non-blocking way. Parameters ---------- blocksize : int The number of bytes to store in each block. Requests are only ever made for ``blocksize``, so this should balance the overhead of making a request against the granularity of the blocks. fetcher : Callable size : int The total size of the file being cached. maxblocks : int The maximum number of blocks to cache for. The maximum memory use for this cache is then ``blocksize * maxblocks``. backgroundrrrvrrrrrrwrrcsZt|||t|||_||_t|j||_t dd|_ d|_ d|_ t |_dS)Nr> max_workers)r7r!rxryrzrwrr}r~r_thread_executor_fetch_future_block_number _fetch_futurerr_fetch_future_lockrr;rr r!s zBackgroundBlockCache.__init__rcCr)Nz r) r^rcrr}r~rrrrrrrr`rrr rds  z!BackgroundBlockCache.__setstate__r"r#rLr%c Cs|durd}|dur |j}||jks||krdS||j}||j}d}d}|jM|jduro|jdus6J|jrRtd|j |j |jd|_d|_nt ||jko]|kn}|ro|j}|j}d|_d|_Wdn1sywY|durtd|j | |t ||dD]}||q|d} |j%|jdur| |j kr|j| s| |_|j|j| d|_Wdn1swY|j||||dS)Nrr'z3BlockCache joined background fetch without waiting.z(BlockCache waiting for background fetch.r>asyncr)rrrrrdonerRrr~rrrrTrzrrsubmitr}r) rr"rLrrfetch_future_block_number fetch_future must_joinrend_block_plus_1rrr r)sv            zBackgroundBlockCache._fetchsyncrlog_infocsV||jkrtd|d|jd||j}||j}td||t||}|S)rrrrz!BlockCache fetching block (%s) %dr)rrrr"rLrr;rr r}s   z!BackgroundBlockCache._fetch_blockrrc Crrrrrrr r%rz BackgroundBlockCache._read_cacherrrrrf)rrre)r)rrrrrr%r)r,r-r.r/rr0r!rrrbrdr)r}rrhrrr;r r~s    Lrz!dict[str | None, type[BaseCache]]cachesFclstype[BaseCache]clobberrrrcCs6|j}|s|tvrtd|dt||t|<dS)z'Register' cache implementation. Parameters ---------- clobber: bool, optional If set to True (default is False) - allow to overwrite existing entry. Raises ------ ValueError zCache with name z is already known: N)rrr)rrrrrr register_cacheSs  r)F)rrrrrr)- __future__rrr{loggingrxrBrrconcurrent.futuresrrtypingrrrrr r r r r2typing_extensionsr rr getLoggerrRrr%rrr1rjrprtrrrrrrr0rcrrrr sZ(    !Q* ]b<R