o Zh?/@sddlmZddlZddlZddlZddlZddlmZddlm Z m Z m Z m Z m Z ddlZddlmZddlmZddlmZeeZGdd d eZdS) ) annotationsN)ThreadPoolExecutor)AnyIterableListOptionalTuple)Document) Embeddings) VectorStorec@seZdZdZ      d.d/ddZ  d0d1ddZ  d0d2d"d#Z d3d4d%d&Ze       d5d6d)d*Z e      d7d8d,d-Z dS)9Clarifaia`Clarifai AI` vector store. To use, you should have the ``clarifai`` python SDK package installed. Example: .. code-block:: python from langchain_community.vectorstores import Clarifai clarifai_vector_db = Clarifai( user_id=USER_ID, app_id=APP_ID, number_of_docs=NUMBER_OF_DOCS, ) Nhttps://api.clarifai.comuser_id Optional[str]app_idnumber_of_docs Optional[int]pattokenapi_basereturnNonec Cs|ptjd}|ptjd}|dus|durtd||_zddlm} Wnty8} ztd| d} ~ ww| ||||||dj|_ dS) aInitialize with Clarifai client. Args: user_id (Optional[str], optional): User ID. Defaults to None. app_id (Optional[str], optional): App ID. Defaults to None. pat (Optional[str], optional): Personal access token. Defaults to None. token (Optional[str], optional): Session token. Defaults to None. number_of_docs (Optional[int], optional): Number of documents to return during vector search. Defaults to None. api_base (Optional[str], optional): API base. Defaults to None. Raises: ValueError: If user ID, app ID or personal access token is not provided. ZCLARIFAI_USER_IDZCLARIFAI_APP_IDNzCould not find CLARIFAI_USER_ID or CLARIFAI_APP_ID in your environment. Please set those env variables with a valid user ID, app IDrSearchXCould not import clarifai python package. Please install it with `pip install clarifai`.)rrtop_krrbase_url) osenvironget ValueError_number_of_docsclarifai.client.searchr ImportErrorZ auth_helper_auth) selfrrrrrrZ_user_idZ_app_idrer(`/var/www/html/lang_env/lib/python3.10/site-packages/langchain_community/vectorstores/clarifai.py__init__#s6zClarifai.__init__texts Iterable[str] metadatasOptional[List[dict]]idsOptional[List[str]]kwargsr List[str]c szddlm}ddlm}Wnty}ztd|d}~wwt|}t|} | dks0Jd|dur>| t|ks>Jd|durNt|t|ksNJd|j|jd d } g} t d| | D]} zd|| | | } |rt|| | | nd|durd d t t| Dn|| | | durgD]}|}| | |qfd d t | D}j |d}| |tdWq_ty}ztd|tWYd}~q_d}~ww| S)a'Add texts to the Clarifai vectorstore. This will push the text to a Clarifai application. Application use a base workflow that create and store embedding for each text. Make sure you are using a base workflow that is compatible with text (such as Language Understanding). Args: texts (Iterable[str]): Texts to add to the vectorstore. metadatas (Optional[List[dict]], optional): Optional list of metadatas. ids (Optional[List[str]], optional): Optional list of IDs. r)Inputs)StructrNz,No texts provided to add to the vectorstore.z1Number of texts and metadatas should be the same.z7Number of text inputs and input ids should be the same.)auth cSsg|]}tjqSr()uuiduuid4hex).0_r(r(r) sz&Clarifai.add_texts..cs0g|]\}}j||r|nddqS)N)Zinput_idZraw_textmetadata)Zget_text_input)r:itextZ batch_idsZbatch_metadatasZ input_objZ meta_listr(r)r<s)ZinputszInput posted successfully.zPost inputs failed: )Zclarifai.client.inputr3Zgoogle.protobuf.struct_pb2r4r$listlenfrom_auth_helperr%rangeupdateappend enumerateZ upload_inputsextendloggerdebug Exceptionwarning traceback print_exc)r&r+r-r/r1r3r4r'ZltextslengthZ batch_sizeZ input_job_idsidxZ batch_textsmetaZ meta_structZ input_batchZ result_iderrorr(r@r) add_textsUsf     zClarifai.add_textsquerystrkfiltersOptional[dict]List[Tuple[Document, float]]c szddlm}ddlm}ddlmWnty%}ztd|d}~ww|p*j}|jj |d} d|ig} |durId |i} | j | | gd } n| j | d } d d | D} t dddfdd fdd | D}dd |D}|S)aRun similarity search with score using Clarifai. Args: query (str): Query text to search for. k (Optional[int]): Number of results to return. If not set, it'll take _number_of_docs. Defaults to None. filter (Optional[Dict[str, str]]): Filter by metadata. Defaults to None. Returns: List[Document]: List of documents most similar to the query text. rr) resources_pb2) json_formatrN)r5rZtext_rawr=)ranksrW)r\cSsg|] }|jD]}|qqSr()hits)r:datahitr(r(r)r<sz9Clarifai.similarity_search_with_score.. ) max_workersr_resources_pb2.HitrTuple[Document, float]c s|jjj}tjj}tj|jjjj |d}|j |_ |j}t d|jdd|jjd|jjd|ddt||d|jfS) N)headersz Score z.2fz for annotation: z off input: z, text: }) page_contentr=)Z MessageToDictinputr^r=dictr%requestsr r?urlapparent_encodingencodingrIrJZscore annotationidr )r_r=hrequestZrequested_text)r[r&r(r)hit_to_documents  z>Clarifai.similarity_search_with_score..hit_to_documentcsg|]}|qSr()submit)r:r_)executorrqr(r)r<scSsg|]}|qSr()result)r:futurer(r(r)r<)r_rbrrc) r#rZclarifai_grpc.grpc.apirZZgoogle.protobufr[r$r"rCr%rTr)r&rTrVrWr1rrZr'rZ search_objZrankZsearch_metadataZsearch_responser]futuresdocs_and_scoresr()rsrqr[r&r)similarity_search_with_scores2      z%Clarifai.similarity_search_with_scoreList[Document]cKs$|j|fd|i|}dd|DS)aGRun similarity search using Clarifai. Args: query: Text to look up documents similar to. k: Number of Documents to return. If not set, it'll take _number_of_docs. Defaults to None. Returns: List of Documents most similar to the query and score for each rVcSsg|]\}}|qSr(r()r:docr;r(r(r)r<rvz.Clarifai.similarity_search..)ry)r&rTrVr1rxr(r(r)similarity_searchszClarifai.similarity_search embeddingOptional[Embeddings]c Ks,|d|||||d| } | j||d| S)aCreate a Clarifai vectorstore from a list of texts. Args: user_id (str): User ID. app_id (str): App ID. texts (List[str]): List of texts to add. number_of_docs (Optional[int]): Number of documents to return during vector search. Defaults to None. pat (Optional[str], optional): Personal access token. Defaults to None. token (Optional[str], optional): Session token. Defaults to None. metadatas (Optional[List[dict]]): Optional list of metadatas. Defaults to None. kwargs: Additional keyword arguments to be passed to the Search. Returns: Clarifai: Clarifai vectorstore. )rrrrr)r+r-Nr()rS) clsr+r}r-rrrrrr1Zclarifai_vector_dbr(r(r) from_textsszClarifai.from_texts documentsc Ks<dd|D} dd|D} |jd||| ||| |d|S)aCreate a Clarifai vectorstore from a list of documents. Args: user_id (str): User ID. app_id (str): App ID. documents (List[Document]): List of documents to add. number_of_docs (Optional[int]): Number of documents to return during vector search. Defaults to None. pat (Optional[str], optional): Personal access token. Defaults to None. token (Optional[str], optional): Session token. Defaults to None. kwargs: Additional keyword arguments to be passed to the Search. Returns: Clarifai: Clarifai vectorstore. cSg|]}|jqSr()rfr:r{r(r(r)r<>z+Clarifai.from_documents..cSrr()r=rr(r(r)r<?r)rrr+rrr-rNr()r) rrr}rrrrrr1r+r-r(r(r)from_documents#szClarifai.from_documents)NNr NNr)rrrrrrrrrrrrrr)NN) r+r,r-r.r/r0r1rrr2) rTrUrVrrWrXr1rrrY)N)rTrUrVrr1rrrz)NNNNNNN)r+r2r}r~r-r.rrrrrrrrrrr1rrr )NNNNNN)rrzr}r~rrrrrrrrrrr1rrr ) __name__ __module__ __qualname____doc__r*rSryr| classmethodrrr(r(r(r)r sD 5 R E )r ) __future__rloggingrrMr7concurrent.futuresrtypingrrrrrriZlangchain_core.documentsr Zlangchain_core.embeddingsr Zlangchain_core.vectorstoresr getLoggerrrIr r(r(r(r)s