o Zh F@sddlmZddlZddlZddlZddlmZddlmZddl m Z m Z m Z m Z mZmZmZddlZddlmZddlmZddlmZdd lmZdd lmZdd lmZdd lm Z e!gd Z"dZ#dddZ$GdddeZ%dS)) annotationsN) ConfigParser)Path)AnyCallableDictIterableListOptionalTuple)Document) Embeddings guard_import) VectorStore)Docstore)InMemoryDocstore)maximal_marginal_relevance)angularZ euclideanZ manhattanZhammingdotrreturnrcCstdS)z1Import annoy if available, otherwise raise error.annoyrrr]/var/www/html/lang_env/lib/python3.10/site-packages/langchain_community/vectorstores/annoy.pydependable_annoy_importsrc@s eZdZdZdYd d ZedZddZ d[d\ddZd]d d!Z #d^d_d(d)Z #d^d`d+d,Z #d^dad.d/Z #d^dbd1d2Z #d^dcd3d4Z #d^ddd5d6Z " 7 8dedfdd?Zeded@d#fdhdFdGZeded@d#fdidHdIZeded@d#fdjdLdMZdkdldSdTZedNdUdmdWdXZdS)nAnnoya`Annoy` vector store. To use, you should have the ``annoy`` python package installed. Example: .. code-block:: python from langchain_community.vectorstores import Annoy db = Annoy(embedding_function, index, docstore, index_to_docstore_id) embedding_functionrindexrmetricstrdocstorerindex_to_docstore_idDict[int, str]cCs"||_||_||_||_||_dS)z%Initialize with necessary components.N)rrrr r!)selfrrrr r!rrr__init__*s   zAnnoy.__init__rOptional[Embeddings]cCsdSNrr#rrr embeddings9szAnnoy.embeddingsNtexts Iterable[str] metadatasOptional[List[dict]]kwargs List[str]cKstd)Nz=Annoy does not allow to add new data once the index is build.)NotImplementedError)r#r)r+r-rrr add_texts>szAnnoy.add_textsidxs List[int]dists List[float]List[Tuple[Document, float]]cCs^g}t||D]%\}}|j|}|j|}t|ts%td|d||||fq|S)aTurns annoy results into a list of documents and scores. Args: idxs: List of indices of the documents in the index. dists: List of distances of the documents in the index. Returns: List of Documents and scores. Could not find document for id , got )zipr!r search isinstancer ValueErrorappend)r#r1r3docsidxdist_iddocrrrprocess_index_resultsHs    zAnnoy.process_index_results embeddingkintsearch_kcC$|jj|||dd\}}|||Sa}Return docs most similar to query. Args: query: Text to look up documents similar to. k: Number of Documents to return. Defaults to 4. search_k: inspect up to search_k nodes which defaults to n_trees * n if not provided Returns: List of Documents most similar to the query and score for each TrHZinclude_distances)rget_nns_by_vectorrB)r#rErFrHr1r3rrr&similarity_search_with_score_by_vector\  z,Annoy.similarity_search_with_score_by_vectordocstore_indexcCrIrJ)rZget_nns_by_itemrB)r#rOrFrHr1r3rrr%similarity_search_with_score_by_indexnrNz+Annoy.similarity_search_with_score_by_indexquerycCs||}||||}|S)a~Return docs most similar to query. Args: query: Text to look up documents similar to. k: Number of Documents to return. Defaults to 4. search_k: inspect up to search_k nodes which defaults to n_trees * n if not provided Returns: List of Documents most similar to the query and score for each )rrM)r#rQrFrHrEr=rrrsimilarity_search_with_scores z"Annoy.similarity_search_with_scoreList[Document]cK||||}dd|DS)aReturn docs most similar to embedding vector. Args: embedding: Embedding to look up documents similar to. k: Number of Documents to return. Defaults to 4. search_k: inspect up to search_k nodes which defaults to n_trees * n if not provided Returns: List of Documents most similar to the embedding. cSg|]\}}|qSrr.0rA_rrr z5Annoy.similarity_search_by_vector..)rM)r#rErFrHr-docs_and_scoresrrrsimilarity_search_by_vectorz!Annoy.similarity_search_by_vectorcKrT)azReturn docs most similar to docstore_index. Args: docstore_index: Index of document in docstore k: Number of Documents to return. Defaults to 4. search_k: inspect up to search_k nodes which defaults to n_trees * n if not provided Returns: List of Documents most similar to the embedding. cSrUrrrVrrrrYrZz4Annoy.similarity_search_by_index..)rP)r#rOrFrHr-r[rrrsimilarity_search_by_indexr]z Annoy.similarity_search_by_indexcKrT)alReturn docs most similar to query. Args: query: Text to look up documents similar to. k: Number of Documents to return. Defaults to 4. search_k: inspect up to search_k nodes which defaults to n_trees * n if not provided Returns: List of Documents most similar to the query. cSrUrrrVrrrrYrZz+Annoy.similarity_search..)rR)r#rQrFrHr-r[rrrsimilarity_searchszAnnoy.similarity_search?fetch_k lambda_multfloatc  sjj||dddfddD}ttj|gtjd|||d}fdd|D}g} |D]!} j| } j| } t | t sKt d | d | | | q/| S) aReturn docs selected using the maximal marginal relevance. Maximal marginal relevance optimizes for similarity to query AND diversity among selected documents. Args: embedding: Embedding to look up documents similar to. fetch_k: Number of Documents to fetch to pass to MMR algorithm. k: Number of Documents to return. Defaults to 4. lambda_mult: Number between 0 and 1 that determines the degree of diversity among the results with 0 corresponding to maximum diversity and 1 to minimum diversity. Defaults to 0.5. Returns: List of Documents selected by maximal marginal relevance. rDFrKcsg|]}j|qSr)rZget_item_vectorrWir'rrrYszAAnnoy.max_marginal_relevance_search_by_vector..)Zdtype)rFrccsg|] }|dkr|qS)rDrre)r1rrrYsr6r7) rrLrnparrayZfloat32r!r r9r:r r;r<) r#rErFrbrcr-r(Z mmr_selectedZselected_indicesr=rfr@rAr)r1r#r'max_marginal_relevance_search_by_vectors&    z-Annoy.max_marginal_relevance_search_by_vectorcKs ||}|j||||d}|S)aReturn docs selected using the maximal marginal relevance. Maximal marginal relevance optimizes for similarity to query AND diversity among selected documents. Args: query: Text to look up documents similar to. k: Number of Documents to return. Defaults to 4. fetch_k: Number of Documents to fetch to pass to MMR algorithm. lambda_mult: Number between 0 and 1 that determines the degree of diversity among the results with 0 corresponding to maximum diversity and 1 to minimum diversity. Defaults to 0.5. Returns: List of Documents selected by maximal marginal relevance. )rc)rri)r#rQrFrbrcr-rEr=rrrmax_marginal_relevance_searchs z#Annoy.max_marginal_relevance_searchdr(List[List[float]]r treesn_jobsc s|tvrtd|dtttd} |stdt|d} | j| |d} t|D] \} } | | | q+| j||dg}t|D]\} }|rM|| ni}| t ||dqCd d t t|Dt fd d t|D}||j | ||S) NzUnsupported distance metric: z. Expected one of rz/embeddings must be provided to build AnnoyIndexrr)rn)Z page_contentmetadatacSsi|] }|ttqSr)ruuiduuid4rerrr 6z Annoy.__from..csi|] \}}||qSrr)rWrfrAZ index_to_idrrrs8rt) INDEX_METRICSr;listrlen AnnoyIndex enumerateZadd_itembuildr<r ranger embed_query)clsr)r(rEr+rrmrnr-rfrrfZembZ documentstextrpr rrurZ__froms.  z Annoy.__fromc Ks(||}|j|||||||fi|S)aConstruct Annoy wrapper from raw documents. Args: texts: List of documents to index. embedding: Embedding function to use. metadatas: List of metadata dictionaries to associate with documents. metric: Metric to use for indexing. Defaults to "angular". trees: Number of trees to use for indexing. Defaults to 100. n_jobs: Number of jobs to use for indexing. Defaults to -1. This is a user friendly interface that: 1. Embeds documents. 2. Creates an in memory docstore 3. Initializes the Annoy database This is intended to be a quick way to get started. Example: .. code-block:: python from langchain_community.vectorstores import Annoy from langchain_community.embeddings import OpenAIEmbeddings embeddings = OpenAIEmbeddings() index = Annoy.from_texts(texts, embeddings) )Zembed_documents _Annoy__from) r~r)rEr+rrmrnr-r(rrr from_texts<s $zAnnoy.from_textstext_embeddingsList[Tuple[str, List[float]]]c Ks:dd|D}dd|D} |j|| |||||fi|S)aConstruct Annoy wrapper from embeddings. Args: text_embeddings: List of tuples of (text, embedding) embedding: Embedding function to use. metadatas: List of metadata dictionaries to associate with documents. metric: Metric to use for indexing. Defaults to "angular". trees: Number of trees to use for indexing. Defaults to 100. n_jobs: Number of jobs to use for indexing. Defaults to -1 This is a user friendly interface that: 1. Creates an in memory docstore with provided embeddings 2. Initializes the Annoy database This is intended to be a quick way to get started. Example: .. code-block:: python from langchain_community.vectorstores import Annoy from langchain_community.embeddings import OpenAIEmbeddings embeddings = OpenAIEmbeddings() text_embeddings = embeddings.embed_documents(texts) text_embedding_pairs = list(zip(texts, text_embeddings)) db = Annoy.from_embeddings(text_embedding_pairs, embeddings) cSg|]}|dqS)rrrWtrrrrYrZz)Annoy.from_embeddings..cSr)rrrrrrYrZ)r) r~rrEr+rrmrnr-r)r(rrrfrom_embeddingses%zAnnoy.from_embeddingsF folder_pathprefaultboolNonecCst|}tj|ddt}|jj|jd|d<|jjt|d|dt |dd}t |j |j |f|Wd d S1sBwYd S) aSave Annoy index, docstore, and index_to_docstore_id to disk. Args: folder_path: folder path to save index, docstore, and index_to_docstore_id to. prefault: Whether to pre-load the index into memory. T)exist_ok)rrANNOY index.annoy)r index.pklwbN)rosmakedirsrrrrsaveropenpickledumpr r!)r#rrpath config_objectfilerrr save_locals "zAnnoy.save_local)allow_dangerous_deserializationrc Cs|stdt|}td}t|dd}t|\}}} Wdn1s(wYt| dd} | dd} |j| | d } | t|d ||j | | ||S) aRLoad Annoy index, docstore, and index_to_docstore_id to disk. Args: folder_path: folder path to load index, docstore, and index_to_docstore_id from. embeddings: Embeddings to use when generating queries. allow_dangerous_deserialization: whether to allow deserialization of the data which involves loading a pickle file. Pickle files can be modified by malicious actors to deliver a malicious payload that results in execution of arbitrary code on your machine. aBThe de-serialization relies loading a pickle file. Pickle files can be modified to deliver a malicious payload that results in execution of arbitrary code on your machine.You will need to set `allow_dangerous_deserialization` to `True` to enable deserialization. If you do this, make sure that you trust the source of the data. For example, if you are loading a file that you created, and know that no one else has modified the file, then this is safe to do. Do not set this to `True` if you are loading a file from an untrusted source (e.g., some random site on the internet.).rrrbNrrrror) r;rrrrloadrGryrr}) r~rr(rrrrr r!rrrrrrr load_locals,   zAnnoy.load_local) rrrrrrr rr!r")rr%r&)r)r*r+r,r-rrr.)r1r2r3r4rr5)rCrD)rEr4rFrGrHrGrr5)rOrGrFrGrHrGrr5)rQrrFrGrHrGrr5) rEr4rFrGrHrGr-rrrS) rOrGrFrGrHrGr-rrrS) rQrrFrGrHrGr-rrrS)rCr`ra) rEr4rFrGrbrGrcrdr-rrrS) rQrrFrGrbrGrcrdr-rrrS)r)r.r(rlrEr r+r,rrrmrGrnrGr-rrr)r)r.rEr r+r,rrrmrGrnrGr-rrr)rrrEr r+r,rrrmrGrnrGr-rrr)F)rrrrrr)rrr(r rrrr)__name__ __module__ __qualname____doc__r$propertyr(r0rBrMrPrRr\r^r_rirj classmethodDEFAULT_METRICrrrrrrrrrrsd          2 %( +r)rr)& __future__rrrrq configparserrpathlibrtypingrrrrr r r numpyrgZlangchain_core.documentsr Zlangchain_core.embeddingsr Zlangchain_core.utilsrZlangchain_core.vectorstoresrZ!langchain_community.docstore.baserZ&langchain_community.docstore.in_memoryrZ&langchain_community.vectorstores.utilsr frozensetrvrrrrrrrs&   $