o TZh @sHUddlZddlZddlZddlZddlmZddlZddlZddlm Z ddl m Z ddl m Z ejdduZerAdd lmZe je je je je jgZee jed <eD]#Zejejvrpejejeurped ejd ejejed dqVe dde de fddZ!dej"de#fddZ$dej"de de fddZ%dddZ&dS)N)List)LocalFileSystem) deprecated) compressionZs3fs) S3FileSystemCOMPRESSION_FILESYSTEMSz*A filesystem protocol was already set for z and will be overwritten.T)clobberzuThis function is deprecated and will be removed in a future version. Please use `fsspec.core.strip_protocol` instead. dataset_pathreturncCsd|vr |dd}|S)z Preprocesses `dataset_path` and removes remote filesystem (e.g. removing `s3://`). Args: dataset_path (`str`): Path (e.g. `dataset/train`) or remote uri (e.g. `s3://my-bucket/dataset/train`) of the dataset directory. z://r)split)r rT/var/www/html/lang_env/lib/python3.10/site-packages/datasets/filesystems/__init__.pyextract_path_from_uri#s rfscCs t|t S)z Checks if `fs` is a remote filesystem. Args: fs (`fsspec.spec.AbstractFileSystem`): An abstract super-class for pythonic file-systems, e.g. `fsspec.filesystem('file')` or [`datasets.filesystems.S3FileSystem`]. ) isinstancer)rrrris_remote_filesystem3s rsrcdstcCs8t|st||||dS|j||dddS)z2 Renames the file `src` in `fs` to `dst`. T) recursiveN)rshutilmoveZ_strip_protocolmv)rrrrrrrename>srcCsBttjdr tjdSdtjjd<dtjjd<ttj_dS)z Clear reference to the loop and thread. This is necessary otherwise HTTPFileSystem hangs in the ML training loop. Only required for fsspec >= 0.9.0 See https://github.com/fsspec/gcsfs/issues/379 reset_lockNr) hasattrfsspecZasynrZiothreadloop threadingLocklockrrrr_reset_fsspec_lockIs   r")r N)' importlibrrwarningstypingrrZ fsspec.asynZfsspec.implementations.localrZutils.deprecation_utilsrrutil find_specZ _has_s3fsZ s3filesystemrZ Bz2FileSystemZGzipFileSystemZ Lz4FileSystemZ XzFileSystemZZstdFileSystemr ZBaseCompressedFileFileSystem__annotations__Zfs_classprotocolregistrywarnZregister_implementationstrrZAbstractFileSystemboolrrr"rrrrs<