o Zht@sdZddlmZddlZddlZddlZddlmZmZm Z m Z m Z m Z m Z ddlZddlmZddlmZddlmZddlmZdd lmZed gZd Zd Zd Zee d j!Z"e#e dj!Z$ej%j!Z&d ddZ'd!ddZ(d!ddZ)d"ddZ*d"ddZ+GdddeZ,dS)#z&Wrapper around TileDB vector database.) annotationsN)AnyDictIterableListMappingOptionalTuple)Document) Embeddings guard_import) VectorStore)maximal_marginal_relevanceZ euclideanZ documentsvectorsuint64float32returnrcCstdtdfS)z@Import tiledb-vector-search if available, otherwise raise error.tiledb.vector_searchtiledbr rr^/var/www/html/lang_env/lib/python3.10/site-packages/langchain_community/vectorstores/tiledb.pydependable_tiledb_importsrgroupstrcC |tjS) Get the URI of the vector index.)VECTOR_INDEX_NAMEurirrrrget_vector_index_uri_from_group#s r cCr)zGet the URI of the documents array from group. Args: group: TileDB group object. Returns: URI of the documents array. )DOCUMENTS_ARRAY_NAMErrrrr"get_documents_array_uri_from_group(s r"rcC|dtS)r/)rrrrrget_vector_index_uri4r&cCr#)z#Get the URI of the documents array.r$)r!r%rrrget_documents_array_uri9r'r(c@seZdZdZdddddddgddZedhddZddeddid'd(Zddd)d*djd,d-Z ddd)d*dkd/d0Z   )dldmd2d3Z   )dldnd4d5Z dd)d6dd7dod9d:Z  ) 6 dpdqd;d<Z  ) 6 dpdrd=d>Zed?dd@dsdGdHZeddedIddJdKdtdSdTZ JdudvdVdWZ   JdwdxdYdZZedded[dIddJfdyd\d]ZeddedIddJdKdzd`daZeedddbd{dcddZd|dedfZdS)}TileDBa2TileDB vector store. To use, you should have the ``tiledb-vector-search`` python package installed. Example: .. code-block:: python from langchain_community import TileDB embeddings = OpenAIEmbeddings() db = TileDB(embeddings, index_uri, metric) NF)vector_index_uridocs_array_uriconfig timestampallow_dangerous_deserialization embeddingr index_urirmetricr+r,r-Optional[Mapping[str, Any]]r.rr/boolkwargscKsV|std||_|j|_||_||_||_tdtd} } | j|d|| |jd} |dkr3|nt | |_ |dkr>|nt | |_ | | |j d} | jd|_| ||_|jdkrs| jjd |j |j|jd | |_n&|jd kr| jjd |j |j|jd | |_Wd d SWd d SWd d S1swYd S) aInitialize with necessary components. Args: allow_dangerous_deserialization: whether to allow deserialization of the data which involves loading data using pickle. data can be modified by malicious actors to deliver a malicious payload that results in execution of arbitrary code on your machine. aTileDB relies on pickle for serialization and deserialization. This can be dangerous if the data is intercepted and/or modified by malicious actors prior to being de-serialized. If you are sure that the data is safe from modification, you can set allow_dangerous_deserialization=True to proceed. Loading of compromised data using pickle can result in execution of arbitrary code on your machine.rrZ ctx_or_configrr* index_typeFLAT)rr-r.IVF_FLATNr) ValueErrorr0Z embed_queryembedding_functionr1r2r-r scope_ctxGroupr r+r"r,closemetagetr8r. flat_indexZ FlatIndex vector_indexivf_flat_indexZ IVFFlatIndex)selfr0r1r2r+r,r-r.r/r5 tiledb_vsr index_grouprrrr__init__Ls`    "zTileDB.__init__rOptional[Embeddings]cCs|jSN)r0rErrr embeddingsszTileDB.embeddings)kfilterscore_thresholdids List[int]scores List[float]rNintrOOptional[Dict[str, Any]]rPfloatList[Tuple[Document, float]]csVtd}g}|j|jd|j|jd}t||D]\} } | dkr$| dkr$q| tkr-| tkr-q|| } | dus=t| ddkrGt d| d| | d } t t | ddd | durnt t| tj} | _|durd d |D}tfd d|Dr|| fq|| fq|fdd|D}|d|S)aTurns TileDB results into a list of documents and scores. Args: ids: List of indices of the documents in the index. scores: List of distances of the documents in the index. k: Number of Documents to return. Defaults to 4. filter (Optional[Dict[str, Any]]): Filter by metadata. Defaults to None. score_threshold: Optional, a floating point value to filter the resulting set of retrieved docs Returns: List of Documents and scores. rr7r.r-rNtextzCould not find document for id z, got metadata) page_contentcSs&i|]\}}|t|ts|gn|qSr) isinstancelist.0keyvaluerrr sz0TileDB.process_index_results..c3s$|] \}}j||vVqdSrJ)r[rAr_) result_docrr s  z/TileDB.process_index_results..cs g|] \}}|kr||fqSrr)r`docscore)rPrr  z0TileDB.process_index_results..)r openr,r.r-zip MAX_UINT64 MAX_FLOAT_32lenr;rAr rpickleloadsnparraytolistastypeuint8tobytesr[itemsallappendr?)rErQrSrNrOrPrdocs docs_arrayidxrgrfZpickled_metadatar[r)rdrPrprocess_index_resultssB   zTileDB.process_index_resultsrNrOfetch_krc Kszd|vr |d}nt}|jjtt|tjgtjfd|dur'|n|i|\}}|j|d|d|||dS)a[Return docs most similar to query. Args: embedding: Embedding vector to look up documents similar to. k: Number of Documents to return. Defaults to 4. filter (Optional[Dict[str, Any]]): Filter by metadata. Defaults to None. fetch_k: (Optional[int]) Number of Documents to fetch before filtering. Defaults to 20. **kwargs: kwargs to be passed to similarity search. Can include: nprobe: Optional, number of partitions to check if using IVF_FLAT index score_threshold: Optional, a floating point value to filter the resulting set of retrieved docs Returns: List of documents most similar to the query text and distance in float for each. Lower score represents more similarity. rPrNNrrQrSrOrNrP) pop MAX_FLOATrCqueryrqrrrtrr}) rEr0rNrOrr5rPdirrr&similarity_search_with_score_by_vectors   z-TileDB.similarity_search_with_score_by_vectorrcKs(||}|j|f|||d|}|S)aReturn docs most similar to query. Args: query: Text to look up documents similar to. k: Number of Documents to return. Defaults to 4. filter (Optional[Dict[str, str]]): Filter by metadata. Defaults to None. fetch_k: (Optional[int]) Number of Documents to fetch before filtering. Defaults to 20. Returns: List of documents most similar to the query text with Distance as float. Lower score represents more similarity. r)r<r)rErrNrOrr5r0rzrrrsimilarity_search_with_scores z#TileDB.similarity_search_with_scoreList[Document]cK(|j|f|||d|}dd|DS)aReturn docs most similar to embedding vector. Args: embedding: Embedding to look up documents similar to. k: Number of Documents to return. Defaults to 4. filter (Optional[Dict[str, str]]): Filter by metadata. Defaults to None. fetch_k: (Optional[int]) Number of Documents to fetch before filtering. Defaults to 20. Returns: List of Documents most similar to the embedding. rcSg|]\}}|qSrrr`rf_rrrrh5z6TileDB.similarity_search_by_vector..)r)rEr0rNrOrr5docs_and_scoresrrrsimilarity_search_by_vectorsz"TileDB.similarity_search_by_vectorcKr)aReturn docs most similar to query. Args: query: Text to look up documents similar to. k: Number of Documents to return. Defaults to 4. filter: (Optional[Dict[str, str]]): Filter by metadata. Defaults to None. fetch_k: (Optional[int]) Number of Documents to fetch before filtering. Defaults to 20. Returns: List of Documents most similar to the query. rcSrrrrrrrrhNrz,TileDB.similarity_search..)r)rErrNrOrr5rrrrsimilarity_search7szTileDB.similarity_search?rNr lambda_multrOrc sd|vr |d}nt}jjtt|tjgtjfd|dur'|n|di|\}} j| d|d||dur@|n|d|d} fdd| D} t tj|gtjd | ||d } g} | D] }| | |qc| S) azReturn docs and their similarity scores selected using the maximal marginal relevance. Maximal marginal relevance optimizes for similarity to query AND diversity among selected documents. Args: embedding: Embedding to look up documents similar to. k: Number of Documents to return. Defaults to 4. fetch_k: Number of Documents to fetch before filtering to pass to MMR algorithm. lambda_mult: Number between 0 and 1 that determines the degree of diversity among the results with 0 corresponding to maximum diversity and 1 to minimum diversity. Defaults to 0.5. Returns: List of Documents and similarity scores selected by maximal marginal relevance and score for each. rPrNNrrcs$g|]\}}j|jgdqSr)r0embed_documentsr\rrKrrrh}szMTileDB.max_marginal_relevance_search_with_score_by_vector..dtype)rNr) rrrCrrqrrrtrr}rry)rEr0rNrrrOr5rPrSindicesresultsrLZ mmr_selectedrrrrKr2max_marginal_relevance_search_with_score_by_vectorPs<    z9TileDB.max_marginal_relevance_search_with_score_by_vectorcKs*|j|f||||d|}dd|DS)aReturn docs selected using the maximal marginal relevance. Maximal marginal relevance optimizes for similarity to query AND diversity among selected documents. Args: embedding: Embedding to look up documents similar to. k: Number of Documents to return. Defaults to 4. fetch_k: Number of Documents to fetch before filtering to pass to MMR algorithm. lambda_mult: Number between 0 and 1 that determines the degree of diversity among the results with 0 corresponding to maximum diversity and 1 to minimum diversity. Defaults to 0.5. Returns: List of Documents selected by maximal marginal relevance. rcSrrrrrrrrhrzBTileDB.max_marginal_relevance_search_by_vector..)r)rEr0rNrrrOr5rrrr'max_marginal_relevance_search_by_vectorsz.TileDB.max_marginal_relevance_search_by_vectorc Ks*||}|j|f||||d|}|S)aReturn docs selected using the maximal marginal relevance. Maximal marginal relevance optimizes for similarity to query AND diversity among selected documents. Args: query: Text to look up documents similar to. k: Number of Documents to return. Defaults to 4. fetch_k: Number of Documents to fetch before filtering (if needed) to pass to MMR algorithm. lambda_mult: Number between 0 and 1 that determines the degree of diversity among the results with 0 corresponding to maximum diversity and 1 to minimum diversity. Defaults to 0.5. Returns: List of Documents selected by maximal marginal relevance. r)r<r) rErrNrrrOr5r0rzrrrmax_marginal_relevance_searchs z$TileDB.max_marginal_relevance_searchT) metadatasr-r8 dimensions vector_typenp.dtyperNonec Csvtdtd}}|j|dz||Wn|jy&} z| d} ~ ww||d} t| j} t| j} |dkrF|jj | |||dn|dkrT|j j | |||d| j | t d|j d d td fttjd } || }|jd tddd}|g}|r|jdtjdd}|||j|dd|d}|j | || j | td| WddS1swYdS)Nrrr6wr9)rrrr-r:)nameidr)rdomainrrZZU1T)rrvarr[F)rsparseZallows_duplicatesattrs)r r=Z group_createZ TileDBErrorr>r&rr(rBcreaterDaddrZDimrlrqrrDomainZAttrruryZ ArraySchemaZArrayr!r?)clsr1r8rrrr-rFrerrrr+docs_uridimdomZ text_attrr metadata_attrZschemarrrrsb         "z TileDB.creater9r)rrQr2r8r-index_timestamptexts List[str]rLList[List[float]]Optional[List[dict]]Optional[List[str]]rc Ks|tvrtd|dtttdtd} } t|tj}|j|||j d|j |du| d| j | d|s@tdt |}t |}|durSd d |D}t|tj}| jjd||||| d krj| nd| d | | |d Z}|durtjt|tjd}tt|D]}|||<qi}t||d<|durtjt|gtd}d }|D]}tjt|tjd||<|d7}q||d<|||<Wdn1swYWdn1swY|d|||| d| S)NzUnsupported distance metric: z. Expected one of rrr)r1r8rrrr-r6z3embeddings must be provided to build a TileDB indexcS g|] }ttdtdqSrrrrandomrandintrlr`rrrrrh?riz!TileDB.__from..r)r8r1 input_vectors external_idsrr-rrrZr[)r0r1r2r-r) INDEX_METRICSr;r^r rqrrrtrrshaperr=r&r(rZ ingestionZingestrjZzerosrnrangeemptyobject frombufferrodumpsru)rrrLr0r1rrQr2r8r-rr5rFrrr+rrArdatarr[rrrZ__froms     %z TileDB.__fromOptional[bool]cKs2t|tj}|jj||dkr|ndddS)amDelete by vector ID or other criteria. Args: ids: List of ids to delete. timestamp: Optional timestamp to delete with. **kwargs: Other keyword arguments that subclasses might use. Returns: Optional[bool]: True if deletion is successful, False otherwise, None if not implemented. rN)rr.T)rqrrrtrrCZ delete_batch)rErQr.r5rrrrdeletees z TileDB.delete Iterable[str]cKs6td}|jt|}|durdd|D}t|tj}tjt |dd} t t |D]} tj|| tj d| | <q/|j j | ||dkrI|nddi} t|| d <|durtjt |gtd} d} |D]} tjt| tjd| | <| d 7} qh| | d <|j|jd |dkr|nd|jd }| ||<||S)aRun more texts through the embeddings and add to the vectorstore. Args: texts: Iterable of strings to add to the vectorstore. metadatas: Optional list of metadatas associated with the texts. ids: Optional ids of each text object. timestamp: Optional timestamp to write new texts with. kwargs: vectorstore specific parameters Returns: List of ids from adding the texts into the vectorstore. rNcSrrrrrrrrhriz$TileDB.add_texts..Orr)rrr.rZrr[rrY)r r0rr^rqrrrtrrrnrrrCZ update_batchrrrorrurjr,r-r?)rErrrQr.r5rrLrrrrzrr[r{rrr add_textszs> zTileDB.add_textsz/tmp/tiledb_arrayc Ks4g} ||} |jd|| |||||||| d | S)aConstruct a TileDB index from raw documents. Args: texts: List of documents to index. embedding: Embedding function to use. metadatas: List of metadata dictionaries to associate with documents. ids: Optional ids of each text object. metric: Metric to use for indexing. Defaults to "euclidean". index_uri: The URI to write the TileDB arrays index_type: Optional, Vector index type ("FLAT", IVF_FLAT") config: Optional, TileDB config index_timestamp: Optional, timestamp to write new texts with. Example: .. code-block:: python from langchain_community import TileDB from langchain_community.embeddings import OpenAIEmbeddings embeddings = OpenAIEmbeddings() index = TileDB.from_texts(texts, embeddings) rrLr0rrQr2r1r8r-rNr)r _TileDB__from) rrr0rrQr2r1r8r-rr5rLrrr from_textss #  zTileDB.from_textstext_embeddingsList[Tuple[str, List[float]]]c KsBdd|D} dd|D} |jd| | |||||||| d | S)aConstruct TileDB index from embeddings. Args: text_embeddings: List of tuples of (text, embedding) embedding: Embedding function to use. index_uri: The URI to write the TileDB arrays metadatas: List of metadata dictionaries to associate with documents. metric: Optional, Metric to use for indexing. Defaults to "euclidean". index_type: Optional, Vector index type ("FLAT", IVF_FLAT") config: Optional, TileDB config index_timestamp: Optional, timestamp to write new texts with. Example: .. code-block:: python from langchain_community import TileDB from langchain_community.embeddings import OpenAIEmbeddings embeddings = OpenAIEmbeddings() text_embeddings = embeddings.embed_documents(texts) text_embedding_pairs = list(zip(texts, text_embeddings)) db = TileDB.from_embeddings(text_embedding_pairs, embeddings) cSg|]}|dqSrrr`trrrrh rz*TileDB.from_embeddings..cSr)rrrrrrrh rrNr)r) rrr0r1rrQr2r8r-rr5rrLrrrfrom_embeddingss % zTileDB.from_embeddings)r2r-r.cKs|d|||||d|S)a}Load a TileDB index from a URI. Args: index_uri: The URI of the TileDB vector index. embedding: Embeddings to use when generating queries. metric: Optional, Metric to use for indexing. Defaults to "euclidean". config: Optional, TileDB config timestamp: Optional, timestamp to use for opening the arrays. )r0r1r2r-r.Nrr)rr1r0r2r-r.r5rrrloadsz TileDB.loadcKs|jjdi||_dS)Nr)rCconsolidate_updates)rEr5rrrr7szTileDB.consolidate_updates)r0r r1rr2rr+rr,rr-r3r.rr/r4r5r)rrI) rQrRrSrTrNrUrOrVrPrWrrX) r0rTrNrUrOrVrrUr5rrrX) rrrNrUrOrVrrUr5rrrX)rMNr~) r0rTrNrUrOrVrrUr5rrr) rrrNrUrOrVrrUr5rrr)r0rTrNrUrrUrrWrOrVr5rrrX)rMr~rN)r0rTrNrUrrUrrWrOrVr5rrr)rrrNrUrrUrrWrOrVr5rrr)r1rr8rrrUrrrr4r-r3rr)rrrLrr0r r1rrrrQrr2rr8rr-r3rrUr5rrr))Nr)rQrr.rUr5rrr)NNr) rrrrrQrr.rUr5rrr)rrr0r rrrQrr2rr1rr8rr-r3rrUr5rrr))rrr0r r1rrrrQrr2rr8rr-r3rrUr5rrr))r1rr0r r2rr-r3r.rr5rrr))r5rrr)__name__ __module__ __qualname____doc__rHpropertyrLrr}rrrrrrr classmethodrDEFAULT_METRICrrrrrrrrrrrr)>sJ =+#   > ' %?Q  725r))rr)rrrr)rrrr)-r __future__rrorsystypingrrrrrrr numpyrqZlangchain_core.documentsr Zlangchain_core.embeddingsr Zlangchain_core.utilsr Zlangchain_core.vectorstoresrZ&langchain_community.vectorstores.utilsr frozensetrrr!rZiinformaxrlZfinform float_inforrr r"r&r(r)rrrrs2 $