o ZhX @sddlmZddlZddlmZddlmZmZm Z m Z m Z ddl m Z ddlmZddlmZmZddlmZdd lmZe eefZed ZGd d d eZdS) ) annotationsN)Path)IteratorLiteralOptionalSequenceUnion)Document)BaseBlobParser) BlobLoaderFileSystemBlobLoader) GenericLoader) get_parserdefaultc sPeZdZdZ d&d'fd d Zd(ddZeddddddddd)d$d%ZZS)*ConcurrentLoaderz%Load and pars Documents concurrently. blob_loaderr blob_parserr num_workersintreturnNonecst||||_dS)N)super__init__r)selfrrr __class__f/var/www/html/lang_env/lib/python3.10/site-packages/langchain_community/document_loaders/concurrent.pyrs zConcurrentLoader.__init__Iterator[Document]c#srtjjjd&fddjD}tj|D] }|EdHqWddS1s2wYdS)z.Load documents lazily with concurrent parsing.) max_workerscsh|] }jj|qSr)submitrZ lazy_parse).0Zblobexecutorrrr )sz-ConcurrentLoader.lazy_load..N) concurrentfuturesThreadPoolExecutorrrZ yield_blobs as_completedresult)rr'futurerr#r lazy_load"s "zConcurrentLoader.lazy_loadz**/[!.]*rNFr)globexcludesuffixes show_progressparserr parser_kwargspath _PathLiker-strr. Sequence[str]r/Optional[Sequence[str]]r0boolr1Union[DEFAULT, BaseBlobParser]r2Optional[dict]c Csbt|||||d} t|tr(|dkr#|jtjkr#|jdi|pi} nt|} n|} || | |dS)aCreate a concurrent generic document loader using a filesystem blob loader. Args: path: The path to the directory to load documents from. glob: The glob pattern to use to find documents. suffixes: The suffixes to use to filter documents. If None, all files matching the glob will be loaded. exclude: A list of patterns to exclude from the loader. show_progress: Whether to show a progress bar or not (requires tqdm). Proxies to the file system loader. parser: A blob parser which knows how to parse blobs into documents num_workers: Max number of concurrent workers to use. parser_kwargs: Keyword arguments to pass to the parser. )r-r.r/r0r)rNr)r isinstancer5rr ) clsr3r-r.r/r0r1rr2rrrrrfrom_filesystem0s  z ConcurrentLoader.from_filesystem)r)rr rr rrrr)rr)r3r4r-r5r.r6r/r7r0r8r1r9rrr2r:rr) __name__ __module__ __qualname____doc__rr, classmethodr= __classcell__rrrrrs r) __future__rconcurrent.futuresr&pathlibrtypingrrrrrZlangchain_core.documentsr Z)langchain_community.document_loaders.baser Z1langchain_community.document_loaders.blob_loadersr r Z,langchain_community.document_loaders.genericr Z5langchain_community.document_loaders.parsers.registryrr5r4DEFAULTrrrrrs