o ZhQ,@sdZddlZddlZddlZddlmZddlmZmZm Z m Z m Z m Z ddl mZddlmZddlmZmZmZmZmZmZmZmZmZmZmZmZeeZ Gdd d eZ!Gd d d eZ"dS) z:Pebblo's safe dataloader is a wrapper for document loadersN)version)AnyDictIterableIteratorListOptional)Document) BaseLoader) BATCH_SIZE_BYTESPLUGIN_VERSIONApp FrameworkIndexedDocumentPebbloLoaderAPIWrappergenerate_size_based_batches get_full_pathget_loader_full_pathget_loader_type get_runtimeget_source_sizec@seZdZUdZdZeed<     d+dddded ed ed ed e ed ede ededefddZ de e fddZ d,ddZdee fddZed,ddZdefddZde efddZd ede e fd!d"Zde e fd#d$Zd%e d&ede fd'd(Zd eddfd)d*ZdS)-PebbloSafeLoaderzkPebblo Safe Loader class is a wrapper around document loaders enabling the data to be scrutinized. F_discover_sentNlocal)classifier_locationanonymize_snippetslangchain_loadernameowner descriptionapi_key load_semanticclassifier_urlrrc Cs|rt|ts td||_tt|_||_tj dp||_ ||_ ||_ t|j|_g|_g|_tt|jdddd} t| |_t|j|_t|_| |j|jd|jdkredt|jini|_||_t|||| d |_|j|jdS) NzMust specify a valid name.ZPEBBLO_LOAD_SEMANTIC.'r)loader source_path source_typesource_path_size)r!rr#r) isinstancestr NameErrorapp_nameuuiduuid4load_idr'osenvirongetr"rr rr(docs docs_with_idtypesplitrr)rr*r batch_sizeloader_details_get_app_detailsappr pb_clientZsend_loader_discover) selfrrrr r!r"r#rrZ loader_namer?b/var/www/html/lang_env/lib/python3.10/site-packages/langchain_community/document_loaders/pebblo.py__init__%s>  "    zPebbloSafeLoader.__init__returncCs|j|_||jS)zxLoad Documents. Returns: list: Documents fetched from load method of the wrapped `loader`. )r'loadr5classify_in_batches)r>r?r?r@rCVs zPebbloSafeLoader.loadc Cst|j|j}g}t|}t|D]6\}}||dk}||_||_|jj|j|j |j |d}| ||j r>| |}n|}||q||_dS)z Classify documents in batches. This is to avoid API timeouts when sending large number of documents. Batches are generated based on the page_content size. )Z loading_endN)rr5r9len enumerate _index_docsr6r=classify_documentsr<r:_add_pebblo_specific_metadatar"_add_semantic_to_docs _unindex_docsextend) r>ZbatchesZprocessed_docsZ total_batchesibatchZ is_last_batchclassified_docsZbatch_processed_docsr?r?r@rDas*      z$PebbloSafeLoader.classify_in_batchesc csz|j}Wnty&}z|jjjd}t|t||d}~ww zt|}Wn ty;g|_ YdSwt |f|_ | |_ |j |j |j|j}|||jra|||_ n||_ |j dVq()zLoad documents in lazy fashion. Raises: NotImplementedError: raised when lazy_load id not implemented within wrapped loader. Yields: list: Documents from loader's lazy loading. z does not implement lazy_load()NTr)r' lazy_loadNotImplementedError __class____name__loggererrornext StopIterationr5listrHr6r=rIr<r:rJr"rKrL)r>Z doc_iteratorexcZerr_strdocclassified_docr?r?r@rQs6           zPebbloSafeLoader.lazy_loadcCs d|_dS)NT)r)clsr?r?r@set_discover_sents z"PebbloSafeLoader.set_discover_sentc Cs:t\}}t|j|j|j|j||ttdtddd}|S)z\Fetch app details. Internal method. Returns: App: App details. Zlangchain_community)rr)rrr r1runtime frameworkZplugin_versionclient_version) rr r.rr r1r rr)r>r`r_r<r?r?r@r;s  z!PebbloSafeLoader._get_app_detailscCddt|jD}|S)z Indexes the documents and returns a list of IndexedDocument objects. Returns: List[IndexedDocument]: A list of IndexedDocument objects with unique IDs. cSs*g|]\}}tddt|i|qS)pb_idr?)rr,dict.0rNr[r?r?r@ sz0PebbloSafeLoader._index_docs..)rGr5)r>r6r?r?r@rHzPebbloSafeLoader._index_docsrPcCsVdd|jD}|D]}|d}||vr||||q dd|D}|S)aF Adds semantic metadata to the given list of documents. Args: classified_docs (Dict): A dictionary of dictionaries containing the classified documents with pb_id as key. Returns: List[Document]: A list of Document objects with added semantic metadata. cSs i|] }|jt|j|jdqS) page_contentmetadata)rcr rjrkrfr[r?r?r@ sz:PebbloSafeLoader._add_semantic_to_docs..rccSsg|]}|qSr?r?rlr?r?r@rgsz:PebbloSafeLoader._add_semantic_to_docs..)r6valuesr4_add_semantic_to_doc)r>rPZ indexed_docsr\Zdoc_idZsemantic_metadata_docsr?r?r@rKs   z&PebbloSafeLoader._add_semantic_to_docscCrb)z Converts a list of IndexedDocument objects to a list of Document objects. Returns: List[Document]: A list of Document objects. cSs g|] \}}t|j|jdqSri)r rjrkrer?r?r@rgsz2PebbloSafeLoader._unindex_docs..)rGr6)r>r5r?r?r@rLrhzPebbloSafeLoader._unindex_docsr[r\cCs8t|di|jd<t|di|jd<|S)a4 Adds semantic metadata to the given document in-place. Args: doc (Document): A Document object. classified_doc (dict): A dictionary containing the classified document. Returns: Document: The Document object with added semantic metadata. entitiesZpebblo_semantic_entitiesZtopicsZpebblo_semantic_topics)rYr4keysrk)r>r[r\r?r?r@ros   z%PebbloSafeLoader._add_semantic_to_docc Cst|jD]4}|j}|jjjdkrt|d|j|d<nt|d|d|j|d<||jidd|d<qdS)z*Add Pebblo specific metadata to documents.ZSharePointLoadersource full_pathZ pb_checksumN) r6rkr'rSrTrr4r(rc)r>rPr[Z doc_metadatar?r?r@rJs    z.PebbloSafeLoader._add_pebblo_specific_metadata)rrNFN)rBN)rT __module__ __qualname____doc__rbool__annotations__r r,rrArr rCrDrrQ classmethodr^r r;rrHrrKrLrdrorJr?r?r?r@rsR     1  "   rc@seZdZdZddddddeedeedeeedeeee fdeeeee fd df d d Z d e e fd d Z d ee fddZdS)PebbloTextLoaderz Loader for text data. Since PebbloSafeLoader is a wrapper around document loaders, this loader is used to load text data directly into Documents. N)rridsrk metadatastextsrrr{rkr|rBcCs"||_||_||_||_||_dS)a Args: texts: Iterable of text data. source: Source of the text data. Optional. Defaults to None. ids: List of unique identifiers for each text. Optional. Defaults to None. metadata: Metadata for all texts. Optional. Defaults to None. metadatas: List of metadata for each text. Optional. Defaults to None. N)r}rrr{rkr|)r>r}rrr{rkr|r?r?r@rAs  zPebbloTextLoader.__init__ccst|jD]9\}}d}|jpi}|jr(|t|jkr(|j|r(||j||jr7|t|jkr7|j|}t|||dVqdS)zi Lazy load text data into Documents. Returns: Iterator of Documents N)idrjrk)rGr}rkr|rFupdater{r )r>rNtextZ_idrkr?r?r@rQ9s  zPebbloTextLoader.lazy_loadcCs g}|D]}||q|S)z` Load text data into Documents. Returns: List of Documents )rQappend)r>Z documentsr[r?r?r@rCIs  zPebbloTextLoader.load)rTrtrurvrr,rrrrrArr rQrCr?r?r?r@rzs*   rz)#rvloggingr2r/importlib.metadatartypingrrrrrrZlangchain_core.documentsr Z)langchain_community.document_loaders.baser Z$langchain_community.utilities.pebblor r r rrrrrrrrr getLoggerrTrUrrzr?r?r?r@s    8 y