o ZhQ@sddlmZddlZddlZddlZddlmZddlmZm Z m Z m Z m Z m Z mZddlZddlmZddlmZddlmZddlmZdd lmZmZdd lmZdd lmZdddZ dddZ!GdddeZ"dS)) annotationsN)Path)AnyCallableDictIterableListOptionalTuple)Document) Embeddings guard_import) VectorStore) AddableMixinDocstore)InMemoryDocstore)DistanceStrategyx np.ndarrayreturncCs$|ttjj|ddddd}|S)z!Normalize vectors to unit length.T)ZaxisZkeepdimsg-q=N)npZclipZlinalgZnorm)rr]/var/www/html/lang_env/lib/python3.10/site-packages/langchain_community/vectorstores/scann.py normalizes rrcCstdS)z= Import `scann` if available, otherwise raise error. scannr rrrrdependable_scann_importsrc@s(eZdZdZddejdfdVddZ  dWdXd!d"Z  dWdYd#d$Z  dWdZd'd(Z d[d\d*d+Z ,  -d]d^d5d6Z ,  -d]d_d9d:Z ,  -d]d`dd?Ze   dbdcdAdBZe  dWdddCdDZe  dWdedFdGZdfdgdKdLZe dfddMdhdOdPZdidRdSZ ,  -d]d_dTdUZdS)jScaNNa`ScaNN` vector store. To use, you should have the ``scann`` python package installed. Example: .. code-block:: python from langchain_community.embeddings import HuggingFaceEmbeddings from langchain_community.vectorstores import ScaNN model_name = "sentence-transformers/all-mpnet-base-v2" db = ScaNN.from_texts( ['foo', 'bar', 'barz', 'qux'], HuggingFaceEmbeddings(model_name=model_name)) db.similarity_search('foo?', k=1) NF embeddingr indexrdocstorerindex_to_docstore_idDict[int, str]relevance_score_fn"Optional[Callable[[float], float]] normalize_L2booldistance_strategyr scann_config Optional[str]c Cs4||_||_||_||_||_||_||_||_dS)z%Initialize with necessary components.N)rr r!r"r(override_relevance_score_fn _normalize_L2Z _scann_config) selfrr r!r"r$r&r(r)rrr__init__3s  zScaNN.__init__texts Iterable[str] embeddingsIterable[List[float]] metadatasOptional[List[dict]]idsOptional[List[str]]kwargsr List[str]cKs&t|jtstd|jdtd)NSIf trying to add texts, the underlying docstore should support adding items, which does notz(Updates are not available in ScaNN, yet.) isinstancer!r ValueErrorNotImplementedError)r-r/r1r3r5r7rrrZ__addHs z ScaNN.__addcKs*|jt|}|j||f||d|S)alRun more texts through the embeddings and add to the vectorstore. Args: texts: Iterable of strings to add to the vectorstore. metadatas: Optional list of metadatas associated with the texts. ids: Optional list of unique IDs. Returns: List of ids from adding the texts into the vectorstore. r3r5)rembed_documentslist _ScaNN__add)r-r/r3r5r7r1rrr add_textsWszScaNN.add_textstext_embeddings!Iterable[Tuple[str, List[float]]]cKsDt|jtstd|jdt|\}}|j||f||d|S)aRun more texts through the embeddings and add to the vectorstore. Args: text_embeddings: Iterable pairs of string and embedding to add to the vectorstore. metadatas: Optional list of metadatas associated with the texts. ids: Optional list of unique IDs. Returns: List of ids from adding the texts into the vectorstore. r9r:r>)r;r!rr<ziprA)r-rCr3r5r7r/r1rrradd_embeddingsls  zScaNN.add_embeddingsOptional[bool]cKstd)a3Delete by vector ID or other criteria. Args: ids: List of ids to delete. **kwargs: Other keyword arguments that subclasses might use. Returns: Optional[bool]: True if deletion is successful, False otherwise, None if not implemented. z*Deletions are not available in ScaNN, yet.)r=)r-r5r7rrrdeletes z ScaNN.delete List[float]kintfilterOptional[Dict[str, Any]]fetch_kList[Tuple[Document, float]]c  sHtj|gtjd}|jrt|}|j||dur|n|\}}g} t|dD]T\} } | dkr0q'|j| } |j | t t sJt d| d|durpdd|D}tfd d |Dro| |d| fq'| |d| fq'|d dur|jtjtjfvrtjntjfd d | D} | d|S)aReturn docs most similar to query. Args: embedding: Embedding vector to look up documents similar to. k: Number of Documents to return. Defaults to 4. filter (Optional[Dict[str, Any]]): Filter by metadata. Defaults to None. fetch_k: (Optional[int]) Number of Documents to fetch before filtering. Defaults to 20. **kwargs: kwargs to be passed to similarity search. Can include: score_threshold: Optional, a floating point value between 0 to 1 to filter the resulting set of retrieved docs Returns: List of documents most similar to the query text and L2 distance in float for each. Lower score represents more similarity. ZdtypeNrrzCould not find document for id z, got cSs&i|]\}}|t|ts|gn|qSr)r;r@.0keyvaluerrr sz@ScaNN.similarity_search_with_score_by_vector..c3s$|] \}}j||vVqdSN)metadatagetrS)docrr s"z?ScaNN.similarity_search_with_score_by_vector..score_thresholdcs"g|] \}}|r||fqSrrrTr[Z similarity)cmpr]rr s z@ScaNN.similarity_search_with_score_by_vector..)rarrayfloat32r,rr Zsearch_batched enumerater"r!searchr;r r<itemsallappendrZr(rMAX_INNER_PRODUCTZJACCARDoperatorgele) r-rrLrNrPr7vectorindicesZscoresdocsjiZ_idr)r_r[r]r&similarity_search_with_score_by_vectorsD       z,ScaNN.similarity_search_with_score_by_vectorquerystrcKs*|j|}|j||f||d|}|S)aReturn docs most similar to query. Args: query: Text to look up documents similar to. k: Number of Documents to return. Defaults to 4. filter (Optional[Dict[str, str]]): Filter by metadata. Defaults to None. fetch_k: (Optional[int]) Number of Documents to fetch before filtering. Defaults to 20. Returns: List of documents most similar to the query text with L2 distance in float. Lower score represents more similarity. rNrP)rZ embed_queryrq)r-rrrLrNrPr7rrnrrrsimilarity_search_with_scores z"ScaNN.similarity_search_with_scoreList[Document]cK(|j||f||d|}dd|DS)aReturn docs most similar to embedding vector. Args: embedding: Embedding to look up documents similar to. k: Number of Documents to return. Defaults to 4. filter (Optional[Dict[str, str]]): Filter by metadata. Defaults to None. fetch_k: (Optional[int]) Number of Documents to fetch before filtering. Defaults to 20. Returns: List of Documents most similar to the embedding. rtcSg|]\}}|qSrrrTr[_rrrr`z5ScaNN.similarity_search_by_vector..)rq)r-rrLrNrPr7docs_and_scoresrrrsimilarity_search_by_vectorsz!ScaNN.similarity_search_by_vectorcKrw)aReturn docs most similar to query. Args: query: Text to look up documents similar to. k: Number of Documents to return. Defaults to 4. filter: (Optional[Dict[str, str]]): Filter by metadata. Defaults to None. fetch_k: (Optional[int]) Number of Documents to fetch before filtering. Defaults to 20. Returns: List of Documents most similar to the query. rtcSrxrrryrrrr`)r{z+ScaNN.similarity_search..)ru)r-rrrLrNrPr7r|rrrsimilarity_searchszScaNN.similarity_searchList[List[float]]cKsHtd}|dtj} |dd} tj|tjd} |rt| } | dur+|j | | } n| tj kr=|j | dd } n |j | dd } g} |durVdd |D}t|D]\}}|rd||ni}| t||d qZtt|}t|t| krtt|d t| d ttt|| }||| ||fd |i|S)Nrr(r)rRZ dot_productZ squared_l2cSsg|]}ttqSr)rsuuiduuid4)rTrzrrrr`Qsz ScaNN.__from..)Z page_contentrYz ids provided for z, documents. Each document should have an id.r&)rrZrEUCLIDEAN_DISTANCErrarbrscann_ops_pybindZcreate_searcherrhbuilderZscore_brute_forcebuildrcrgr dictlen ExceptionrrEvalues)clsr/r1rr3r5r&r7rr(r)rlr Z documentsrptextrYZ index_to_idr!rrrZ__from+sR    z ScaNN.__fromcKs&||}|j|||f||d|S)aNConstruct ScaNN wrapper from raw documents. This is a user friendly interface that: 1. Embeds documents. 2. Creates an in memory docstore 3. Initializes the ScaNN database This is intended to be a quick way to get started. Example: .. code-block:: python from langchain_community.vectorstores import ScaNN from langchain_community.embeddings import OpenAIEmbeddings embeddings = OpenAIEmbeddings() scann = ScaNN.from_texts(texts, embeddings) r>)r? _ScaNN__from)rr/rr3r5r7r1rrr from_textsgs zScaNN.from_textsList[Tuple[str, List[float]]]cKs8dd|D}dd|D}|j|||f||d|S)aConstruct ScaNN wrapper from raw documents. This is a user friendly interface that: 1. Embeds documents. 2. Creates an in memory docstore 3. Initializes the ScaNN database This is intended to be a quick way to get started. Example: .. code-block:: python from langchain_community.vectorstores import ScaNN from langchain_community.embeddings import OpenAIEmbeddings embeddings = OpenAIEmbeddings() text_embeddings = embeddings.embed_documents(texts) text_embedding_pairs = list(zip(texts, text_embeddings)) scann = ScaNN.from_embeddings(text_embedding_pairs, embeddings) cSg|]}|dqS)rrrTtrrrr`r{z)ScaNN.from_embeddings..cSr)rrrrrrr`r{r>)r)rrCrr3r5r7r/r1rrrfrom_embeddingsszScaNN.from_embeddings folder_path index_nameNonecCst|}|dj|d}|jddd|jt|t|dj|dd}t|j |j f|WddS1s.cs g|] \}}|kr||fqSrrr^)r]rrr`0s )poprr<ru)r-rrrLrNrPr7r|Zdocs_and_rel_scoresr)r$r]r(_similarity_search_with_relevance_scoress.   z.ScaNN._similarity_search_with_relevance_scores)rr r rr!rr"r#r$r%r&r'r(rr)r*)NN) r/r0r1r2r3r4r5r6r7rrr8) r/r0r3r4r5r6r7rrr8) rCrDr3r4r5r6r7rrr8rX)r5r6r7rrrG)rINrJ) rrKrLrMrNrOrPrMr7rrrQ) rrrsrLrMrNrOrPrMr7rrrQ) rrKrLrMrNrOrPrMr7rrrv) rrrsrLrMrNrOrPrMr7rrrv)NNF)r/r8r1rrr r3r4r5r6r&r'r7rrr) r/r8rr r3r4r5r6r7rrr) rCrrr r3r4r5r6r7rrr)r )rrsrrsrr) rrsrr rrsrr'r7rrr)rr)__name__ __module__ __qualname____doc__rrr.rArBrFrHrqrur}r~ classmethodrrrrrrrrrrrr!sr     C "   ;# & 5r)rrrr)rr)# __future__rrirrpathlibrtypingrrrrrr r numpyrZlangchain_core.documentsr Zlangchain_core.embeddingsr Zlangchain_core.utilsrZlangchain_core.vectorstoresrZ!langchain_community.docstore.baserrZ&langchain_community.docstore.in_memoryrZ&langchain_community.vectorstores.utilsrrrrrrrrs"  $