o Zh;@sddlmZddlZddlmZddlmZddlmZm Z m Z m Z m Z ddl ZddlmZddlmZddlmZdd lmZdd lmZeeZGd d d eZdS) ) annotationsN)deepcopy)Enum)AnyIterableListOptionalTuple)Document) Embeddings)run_in_executor) VectorStore)maximal_marginal_relevancec@seZdZdZ dPdQd d ZedRddZ   dSdTddZe       dUdVd!d"Z Gd#d$d$e Z d%e j dfdWd,d-Z d%e j dfdXd/d0Zd%e j dfdYd2d3Zd%e j dfdZd4d5Z % 6 7d[dd8d\dd]d^dBdCZd_dFdGZd`dIdJZdadbdLdMZ dadbdNdOZdS)cRockseta`Rockset` vector store. To use, you should have the `rockset` python package installed. Note that to use this, the collection being used must already exist in your Rockset instance. You must also ensure you use a Rockset ingest transformation to apply `VECTOR_ENFORCE` on the column being used to store `embedding_key` in the collection. See: https://rockset.com/blog/introducing-vector-search-on-rockset/ for more details Everything below assumes `commons` Rockset workspace. Example: .. code-block:: python from langchain_community.vectorstores import Rockset from langchain_community.embeddings.openai import OpenAIEmbeddings import rockset # Make sure you use the right host (region) for your Rockset instance # and APIKEY has both read-write access to your collection. rs = rockset.RocksetClient(host=rockset.Regions.use1a1, api_key="***") collection_name = "langchain_demo" embeddings = OpenAIEmbeddings() vectorstore = Rockset(rs, collection_name, embeddings, "description", "description_embedding") commonsclientr embeddingsr collection_namestrtext_key embedding_key workspacecCszddlm}Wn tytdwt||s"tdt|||_||_||_||_ ||_ ||_ z |j dWdSt yGYdSw)aNInitialize with Rockset client. Args: client: Rockset client object collection: Rockset collection to insert docs / query embeddings: Langchain Embeddings object to use to generate embedding for given text. text_key: column in Rockset collection to use to store the text embedding_key: column in Rockset collection to use to store the embedding. Note: We must apply `VECTOR_ENFORCE()` on this column via Rockset ingest transformation. r) RocksetClient]Could not import rockset client python package. Please install it with `pip install rockset`.z;client should be an instance of rockset.RocksetClient, got Z langchainN)rocksetr ImportError isinstance ValueErrortype_client_collection_name _embeddings _text_key_embedding_key _workspaceZset_applicationAttributeError)selfrrrrrrrr'a/var/www/html/lang_env/lib/python3.10/site-packages/langchain_community/vectorstores/rocksetdb.py__init__1s0   zRockset.__init__returncCs|jSN)r!r&r'r'r(raszRockset.embeddingsN texts Iterable[str] metadatasOptional[List[dict]]idsOptional[List[str]] batch_sizeintkwargs List[str]c Ksg}g}t|D]D\}} t||kr|||7}g}i} |r+t||kr+t||} |r9t||kr9||| d<| | |j<|j| | |j<|| qt|dkr\|||7}g}|S)aRun more texts through the embeddings and add to the vectorstore Args: texts: Iterable of strings to add to the vectorstore. metadatas: Optional list of metadatas associated with the texts. ids: Optional list of ids to associate with the texts. batch_size: Send documents in batches to rockset. Returns: List of ids from adding the texts into the vectorstore. _idr) enumeratelen_write_documents_to_rocksetrr"r! embed_queryr#append) r&r.r0r2r4r6batchZ stored_idsitextdocr'r'r( add_textses$      zRockset.add_texts embeddingc KsX|dusJd|sJd|sJd|sJd||||||} | |||| | S)znCreate Rockset wrapper with existing texts. This is intended as a quicker way to get started. NzRockset Client cannot be NonezCollection name cannot be emptyzText key name cannot be emptyzEmbedding key cannot be empty)rB) clsr.rDr0rrrrr2r4r6rr'r'r( from_textss   zRockset.from_textsc@s"eZdZdZdZdZd ddZdS) zRockset.DistanceFunction COSINE_SIMEUCLIDEAN_DIST DOT_PRODUCTr*rcCs|jdkrdSdS)NrHASCZDESC)valuer,r'r'r(order_bys z!Rockset.DistanceFunction.order_byN)r*r)__name__ __module__ __qualname__rGrHrIrLr'r'r'r(DistanceFunctions rPqueryk distance_func where_str Optional[str]List[Tuple[Document, float]]cK |j|j||||fi|S)aPerform a similarity search with Rockset Args: query (str): Text to look up documents similar to. distance_func (DistanceFunction): how to compute distance between two vectors in Rockset. k (int, optional): Top K neighbors to retrieve. Defaults to 4. where_str (Optional[str], optional): Metadata filters supplied as a SQL `where` condition string. Defaults to None. eg. "price<=70.0 AND brand='Nintendo'" NOTE: Please do not let end-user to fill this and always be aware of SQL injection. Returns: List[Tuple[Document, float]]: List of documents with their relevance score )1similarity_search_by_vector_with_relevance_scoresr!r<r&rRrSrTrUr6r'r'r('similarity_search_with_relevance_scoress z/Rockset.similarity_search_with_relevance_scoresList[Document]cKrX)zaSame as `similarity_search_with_relevance_scores` but doesn't return the scores. )similarity_search_by_vectorr!r<rZr'r'r(similarity_searchs zRockset.similarity_search List[float]cKs&|j||||fi|}dd|DS)zZAccepts a query_embedding (vector), and returns documents with similar embeddings.cSsg|]\}}|qSr'r').0rA_r'r'r( z7Rockset.similarity_search_by_vector..)rY)r&rDrSrTrUr6Zdocs_and_scoresr'r'r(r]s  z#Rockset.similarity_search_by_vectorc Ks:d}d|vr |d}||||||}z |jjjd|id}Wnty9} ztd| gWYd} ~ Sd} ~ wwg} |jD][} i} t| t sQJd t | | D]9\}} ||j krqt| tsnJd |j t | | }qU|d krt| tsJd t | | }qU|d vr| | |<qU| t|| d |fq?| S) z|Accepts a query_embedding (vector), and returns documents with similar embeddings along with their relevance scores.Texclude_embeddingsrR)sqlz$Exception when querying Rockset: %s Nz;document should be of type `dict[str,Any]`. But found: `{}`zIpage content stored in column `{}` must be of type `str`. But found: `{}`distzDComputed distance between vectors must of type `float`. But found {})r8Z _event_time_meta) page_contentmetadata)_build_query_sqlrZQueriesrR Exceptionloggererrorresultsrdictformatritemsr"rfloatr=r )r&rDrSrTrUr6rdZq_strZquery_responseeZ finalResultZdocumentrivrhZscorer'r'r(rYsV          z9Rockset.similarity_search_by_vector_with_relevance_scores?)rUfetch_k lambda_multrrc  sxj|}j|f||dd|fddD}tt||||d} | D] } | jj=q)fdd| DS)aReturn docs selected using the maximal marginal relevance. Maximal marginal relevance optimizes for similarity to query AND diversity among selected documents. Args: query: Text to look up documents similar to. k: Number of Documents to return. Defaults to 4. fetch_k: Number of Documents to fetch to pass to MMR algorithm. distance_func (DistanceFunction): how to compute distance between two vectors in Rockset. lambda_mult: Number between 0 and 1 that determines the degree of diversity among the results with 0 corresponding to maximum diversity and 1 to minimum diversity. Defaults to 0.5. where_str: where clause for the sql query Returns: List of Documents selected by maximal marginal relevance. F)rSrUrdcsg|]}|jjqSr')rir#)r`rAr,r'r(rbVsz9Rockset.max_marginal_relevance_search..)rxrScsg|]}|qSr'r'r`r?) initial_docsr'r(rbdrc)r!r<r]rnparrayrir#) r&rRrSrwrxrUr6query_embeddingrZselected_indicesr?r')rzr&r(max_marginal_relevance_search0s(  z%Rockset.max_marginal_relevance_searchTr}rdboolc Csdtt|}|jd|jd|d}|rd|dnd}|r(d|jd nd}d |d |d |jd |jd|d|dt|dS)zABuilds Rockset SQL query to query similar vectors to query_vector,(z, [z ]) as distzWHERE  rCz EXCEPT(z),zSELECT * z FROM .zORDER BY dist z LIMIT )joinmaprrKr#r$r rL) r&r}rTrSrUrdZq_embedding_strZ distance_strZselect_embeddingr'r'r(rjhs, zRockset._build_query_sqlr> List[dict]cCs(|jjj|j||jd}dd|jDS)NZ collectiondatarcSsg|]}|jqSr')r8)r`Z doc_statusr'r'r(rbsz7Rockset._write_documents_to_rockset..)r DocumentsZ add_documentsr r$r)r&r>Z add_doc_resr'r'r(r;s z#Rockset._write_documents_to_rocksetNonecsRzddlmWn tytdw|jjj|jfdd|D|jddS)z1Delete a list of docs from the Rockset collectionrDeleteDocumentsRequestDatarcsg|]}|dqS))idr'ryrr'r(rbsz(Rockset.delete_texts..rN)Zrockset.modelsrrrrZdelete_documentsr r$)r&r2r'rr( delete_textss  zRockset.delete_textsOptional[bool]c KsPz|durg}||WdSty'}z td|WYd}~dSd}~ww)Nz.Exception when deleting docs from Rockset: %s FT)rrkrlrm)r&r2r6rsr'r'r(deletes  zRockset.deletecstd|j|fi|IdHSr+)r r)r&r2r6r'r'r(adeleteszRockset.adelete)r) rrrr rrrrrrrr)r*r )NNr-) r.r/r0r1r2r3r4r5r6rr*r7)NNrCrCrCNr-)r.r7rDr r0r1rrrrrrrrr2r3r4r5r6rr*r) rRrrSr5rTrPrUrVr6rr*rW) rRrrSr5rTrPrUrVr6rr*r\) rDr_rSr5rTrPrUrVr6rr*r\) rDr_rSr5rTrPrUrVr6rr*rW)rQrurv)rRrrSr5rwr5rxrrrUrVr6rr*r\)rQNT) r}r_rTrPrSr5rUrVrdrr*r)r>rr*r7)r2r7r*rr+)r2r3r6rr*r)rMrNrO__doc__r)propertyrrB classmethodrFrrPrGr[r^r]rYr~rjr;rrrr'r'r'r(rsf$ 0  ( $   :<    r) __future__rloggingcopyrenumrtypingrrrrr numpyr{Zlangchain_core.documentsr Zlangchain_core.embeddingsr Zlangchain_core.runnablesr Zlangchain_core.vectorstoresr Z&langchain_community.vectorstores.utilsr getLoggerrMrlrr'r'r'r(s