o Zhda@sddlmZddlZddlZddlZddlZddlmZmZm Z m Z m Z m Z m Z ddlZddlmZddlmZddlmZddlmZddlmZd Zdd d ZdddZGdddeZdS)) annotationsN)AnyCallableDictIterableListOptionalType)Document) Embeddings guard_import) VectorStore)maximal_marginal_relevancereturnrcCstdS)zImport lancedb package.lancedbr rr_/var/www/html/lang_env/lib/python3.10/site-packages/langchain_community/vectorstores/lancedb.pyimport_lancedbsrfilterDict[str, str]strcCsddd|DS)z2Converts a dict filter to a LanceDB filter string.z AND cSs g|] \}}|d|dqS)z = ''r).0kvrrr s z#to_lance_filter..)joinitems)rrrrto_lance_filtersr c@speZdZdZdddddddddddd ddefdrddZdsdtd&d'Zedud(d)Z  dvdwd2d3Z dxdyd7d8Z   9 :  ; dzd{dDdEZ d|dGdHZ  dvd}dJdKZ   d~ddOdPZddRdSZ   d~ddVdWZ   d~ddXdYZ  dvddZd[Z    ddd^d_Z  ` a dddedfZ  ` a dddgdhZe           dddldmZ     dddpdqZdS)LanceDBay`LanceDB` vector store. To use, you should have ``lancedb`` python package installed. You can install it with ``pip install lancedb``. Args: connection: LanceDB connection to use. If not provided, a new connection will be created. embedding: Embedding to use for the vectorstore. vector_key: Key to use for the vector in the database. Defaults to ``vector``. id_key: Key to use for the id in the database. Defaults to ``id``. text_key: Key to use for the text in the database. Defaults to ``text``. table_name: Name of the table to use. Defaults to ``vectorstore``. api_key: API key to use for LanceDB cloud database. region: Region to use for LanceDB cloud database. mode: Mode to use for adding data to the table. Valid values are ``append`` and ``overwrite``. Defaults to ``overwrite``. Example: .. code-block:: python vectorstore = LanceDB(uri='/lancedb', embedding_function) vectorstore.add_texts(['text1', 'text2']) result = vectorstore.similarity_search('text1') Nz /tmp/lancedbvectoridtext vectorstore overwritel2 connection Optional[Any] embeddingOptional[Embeddings]uri Optional[str] vector_keyid_keytext_key table_nameapi_keyregionmodetabledistancererankerrelevance_score_fn"Optional[Callable[[float], float]]limitintcCstd}td|j_||_||_||_||_|dkr!|p tdnd|_ | |_ | |_ | |_ ||_ ||_d|_t| |jjrA| |_n | durId|_ntdt|tr`|j dur`|dr`td|jduritd t||jjrt||_n:t|t|jjfrtd |j dur|||_n!t|tr|dr|j||j |j d |_n |||_td | durzt| |jj|jjjfsJ| |_t | d r| j!nd|_"WdSt#ytdw|j$|dd|_dS)z$Initialize with Lance DB vectorstorerzlancedb.remote.tableZ LANCE_API_KEYNz9`reranker` has to be a lancedb.rerankers.Reranker object.zdb://z&API key is required for LanceDB cloud.z#embedding object should be providedzs`connection` has to be a lancedb.db.LanceDBConnection object. `lancedb.db.LanceTable` is deprecated.)r2r3z[api key provided with local uri. The data will be stored locallynameZ remote_tablezj`table` has to be a lancedb.db.LanceTable or lancedb.remote.table.RemoteTable object.T) set_default)%r remoter5 _embedding _vector_key_id_key _text_keyosgetenvr2r3r4r6override_relevance_score_fnr: _fts_index isinstanceZ rerankersZReranker _reranker ValueErrorr startswithdbZLanceDBConnection _connectionZ LanceTableconnectwarningswarnZ RemoteTable_tablehasattrr= _table_nameAssertionError get_table)selfr(r*r,r.r/r0r1r2r3r4r5r6r7r8r:rrrr__init__:st         zLanceDB.__init__Fresultsrscoreboolrcsjj}d|vr dn d|vrdndd|vdus|s-fddttDSrA|rCfddttDSdSdS)NZ _distanceZ_relevance_scoremetadatacs:g|]}tj|rd|nidqSr[)Z page_contentr[r rCZas_pyridx) has_metadatarXrVrrrsz+LanceDB.results_to_docs..csJg|]!}tj|rd|nid|fqSr\r]r^r`rXZ score_colrVrrrs )Zschemanamesrangelen)rVrXrYcolumnsrrarresults_to_docss    zLanceDB.results_to_docscCs|jS)N)r@rVrrr embeddingsszLanceDB.embeddingstexts Iterable[str] metadatasOptional[List[dict]]idsOptional[List[str]]kwargs List[str]c Ksg}|p dd|D}|jt|}t|D]&\}}||} |r%||nd||i} ||j| |j|||j|d| iq|} | durS|j j |j |d} | |_ n|j dura| j||jdn| |d|_|S)aTurn texts into embedding and add it to the database Args: texts: Iterable of strings to add to the vectorstore. metadatas: Optional list of metadatas associated with the texts. ids: Optional list of ids to associate with the texts. ids: Optional list of ids to associate with the texts. Returns: List of ids of the added texts. cSg|]}ttqSrruuiduuid4r_rrrrz%LanceDB.add_texts..r#r[Ndata)r4)r@Zembed_documentslist enumerateappendrArBrCrUrM create_tablerSrQr2addr4rG) rVrirkrmrodocsrhr_r$r*r[tblrrr add_textss,    zLanceDB.add_textsr=r>Optional[bool]cCsJ|dur|r ||_|j}n|}n|j}z|j|WSty$YdSw)a  Fetches a table object from the database. Args: name (str, optional): The name of the table to fetch. Defaults to None and fetches current table object. set_default (bool, optional): Sets fetched table as the default table. Defaults to False. Returns: Any: The fetched table object. Raises: ValueError: If the specified table is not found in the database. N)rSrMZ open_table Exception)rVr=r>_namerrrrUs zLanceDB.get_table`L2col_name vector_colnum_partitions Optional[int]num_sub_vectorsindex_cache_sizemetricNonec Cs@||}|r|j|||||ddS|r||dStd)aO Create a scalar(for non-vector cols) or a vector index on a table. Make sure your vector column has enough data before creating an index on it. Args: vector_col: Provide if you want to create index on a vector column. col_name: Provide if you want to create index on a non-vector column. metric: Provide the metric to use for vector index. Defaults to 'L2' choice of metrics: 'L2', 'dot', 'cosine' num_partitions: Number of partitions to use for the index. Defaults to 256. num_sub_vectors: Number of sub-vectors to use for the index. Defaults to 96. index_cache_size: Size of the index cache. Defaults to None. name: Name of the table to create index on. Defaults to None. Returns: None )rvector_column_namerrrz%Provide either vector_col or col_nameN)rU create_indexZcreate_scalar_indexrJ) rVrrrrrrr=rrrrr s  zLanceDB.create_indexrcCsBt|d}t|dWdS1swYdS)z!Get base64 string from image URI.rbzutf-8N)openbase64 b64encodereaddecode)rVr,Z image_filerrr encode_image3s $zLanceDB.encode_imageurisc s}fdd|D}|durdd|D}d}jdur-tjdr-jj|d}ntdg}t|D]$\} } |rA|| nd|| i} |j| j|| j || d | iq7|durnj j j |d }|_ |S|||S) asRun more images through the embeddings and add to the vectorstore. Args: uris List[str]: File path to the image. metadatas (Optional[List[dict]], optional): Optional list of metadatas. ids (Optional[List[str]], optional): Optional list of IDs. Returns: List[str]: List of IDs of the added images. csg|]}j|dqS))r,)r)rr,rgrrrLrwz&LanceDB.add_images..NcSrqrrrrurrrrOrw embed_image)rzEembedding object should be provided and must have embed_image method.r#r[rx)rUr@rRrrJr{r|rArBrCrMr}rSrQr~) rVrrkrmrorZ b64_textsrhryr_Zembr[rrgr add_images8s4   zLanceDB.add_imagesqueryrrc Ks|dur|j}||}t|trt|}|dd}|dd}|d} r<|j||jd|| j ||d} n|j||jd|j ||d} |dkr\|j dur\| j |j d | } t | d krktd | S) N prefilterF query_typer"metrics)rr)rhybrid)r7rzNo results found for the query.)r:rUrHdictr getsearchrArwhererIZrerankZto_arrowrdrOrP) rVrrrr=rorrrrZ lance_queryrrrr_queryls2      zLanceDB._queryCallable[[float], float]cCsN|jr|jS|jdkr|jS|jdkr|jS|jdkr|jStd|jd)a8 The 'correct' relevance function may differ depending on a few things, including: - the distance / similarity metric used by the VectorStore - the scale of your embeddings (OpenAI's are unit normed. Many others are not!) - embedding dimensionality - etc. Zcosiner'ipzANo supported normalization function for distance metric of type: z=.Consider providing relevance_score_fn to Chroma constructor.)rFr6Z_cosine_relevance_score_fnZ_euclidean_relevance_score_fnZ%_max_inner_product_relevance_score_fnrJrgrrr_select_relevance_score_fns    z"LanceDB._select_relevance_score_fn List[float]Optional[Dict[str, str]]cKs>|dur|j}|j||f||d|}|j||dddS)zD Return documents most similar to the query vector. Nrr=rYFrY)r:rrfpop)rVr*rrr=roresrrrsimilarity_search_by_vectors z#LanceDB.similarity_search_by_vectorc s@|dur|j}||j||fddi|}fdd|DS)zZ Return documents most similar to the query vector with relevance scores. NrYTcs g|] \}}|t|fqSr)float)rdocrYr8rrrszMLanceDB.similarity_search_by_vector_with_relevance_scores..)r:rr)rVr*rrr=roZdocs_and_scoresrrr1similarity_search_by_vector_with_relevance_scoress  z9LanceDB.similarity_search_by_vector_with_relevance_scoresc Ks|dur|j}|dd}|dd}|dd}|jdur"td|dks*|d krk|jdurg|jdurg||}|j|jdd |_|d krQ|j |} | |f} n|} |j | |f||d |} |j | |d St d |j |} |j | |fd|i|} |j | |d S)zAReturn documents most similar to the query with relevance scores.NrYTr=rr"z4search needs an emmbedding function to be specified.ftsr)replacerrz?Full text/ Hybrid search is not supported in LanceDB Cloud yet.r) r:rr@rJr2rGrUZcreate_fts_indexrC embed_queryrrfNotImplementedError) rVrrrrorYr=rrr*rrrrrsimilarity_search_with_scores.        z$LanceDB.similarity_search_with_scorerList[Document]c Ks"|jd|||||dd|}|S)apReturn documents most similar to the query Args: query: String to query the vectorstore with. k: Number of documents to return. filter (Optional[Dict]): Optional filter arguments sql_filter(Optional[string]): SQL filter to apply to the query. prefilter(Optional[bool]): Whether to apply the filter prior to the vector search. Raises: ValueError: If the specified table is not found in the database. Returns: List of documents most similar to the query. F)rrr=rrrYNr)r)rVrrr=rrrorrrrsimilarity_searchs  zLanceDB.similarity_search?fetch_k lambda_multrc KsD|dur|j}|jdurtd|j|}|j|||||d}|S)a?Return docs selected using the maximal marginal relevance. Maximal marginal relevance optimizes for similarity to query AND diversity among selected documents. Args: query: Text to look up documents similar to. k: Number of Documents to return. Defaults to 4. fetch_k: Number of Documents to fetch to pass to MMR algorithm. lambda_mult: Number between 0 and 1 that determines the degree of diversity among the results with 0 corresponding to maximum diversity and 1 to minimum diversity. Defaults to 0.5. filter (Optional[Dict[str, str]]): Filter by metadata. Defaults to None. Returns: List of Documents selected by maximal marginal relevance. NzBFor MMR search, you must specify an embedding function oncreation.)rr)r:r@rJr'max_marginal_relevance_search_by_vector) rVrrrrrror*rrrrmax_marginal_relevance_searchs  z%LanceDB.max_marginal_relevance_searchc  sf|jd|||d|}ttj|tjd|d|p|j|d||}fddt|D} | S) aHReturn docs selected using the maximal marginal relevance. Maximal marginal relevance optimizes for similarity to query AND diversity among selected documents. Args: embedding: Embedding to look up documents similar to. k: Number of Documents to return. Defaults to 4. fetch_k: Number of Documents to fetch to pass to MMR algorithm. lambda_mult: Number between 0 and 1 that determines the degree of diversity among the results with 0 corresponding to maximum diversity and 1 to minimum diversity. Defaults to 0.5. filter (Optional[Dict[str, str]]): Filter by metadata. Defaults to None. Returns: List of Documents selected by maximal marginal relevance. )rrr)Zdtyper")rrcsg|] \}}|vr|qSrr)rirZ mmr_selectedrrrmszCLanceDB.max_marginal_relevance_search_by_vector..Nr) rrnparrayZfloat32Z to_pylistr:rfr{) rVr*rrrrrorX candidatesZselected_resultsrrrrCs   z/LanceDB.max_marginal_relevance_search_by_vectorcls Type[LanceDB]r cKs:td||||||| | | | | |d |}|j||d|S)N) r(r*r.r/r0r1r2r3r4r6r7r8)rkr)r!r)rrir*rkr(r.r/r0r1r2r3r4r6r7r8roinstancerrr from_textsps$ zLanceDB.from_texts delete_all drop_columnscKs||}|r||dS|r!||jdd|dS|r3|jdur,td||dS|r<|ddStd)a Allows deleting rows by filtering, by ids or drop columns from the table. Args: filter: Provide a string SQL expression - "{col} {operation} {value}". ids: Provide list of ids to delete from the table. drop_columns: Provide list of columns to drop from the table. delete_all: If True, delete all rows from the table. z in ('{}'),Nz;Column operations currently not supported in LanceDB Cloud.truez6Provide either filter, ids, drop_columns or delete_all) rUdeleterBformatrr2rrrJ)rVrmrrrr=rorrrrrs " zLanceDB.delete)r(r)r*r+r,r-r.r-r/r-r0r-r1r-r2r-r3r-r4r-r5r)r6r-r7r)r8r9r:r;)F)rXrrYrZrr)rr+)NN) rirjrkrlrmrnrorrrp)NF)r=r-r>rrr)NNrrNrN)rr-rr-rrrrrrrr-r=r-rr)r,rrr) rrprkrlrmrnrorrrp)NNN) rrrrrr)r=r-rorrr)rr) r*rrrrrr=r-rorrr) rrrrrrrorrr)NNNF)rrrrr=r-rr)rrrorrr)NrrN)rrrrrr;rrrrrorrr)r*rrrrr;rrrrrorrr) NNr"r#r$r%NNr&r'NN)"rrrirpr*r rkrlr(r)r.r-r/r-r0r-r1r-r2r-r3r-r4r-r6r-r7r)r8r9rorrr!)NNNNN)rmrnrrrr-rrnr=r-rorrr)__name__ __module__ __qualname____doc__ DEFAULT_KrWrfpropertyrhrrUrrrrrrrrrrr classmethodrrrrrrr!s X"  1 # * 7 &   +   / -'r!)rr)rrrr) __future__rrrDrsrOtypingrrrrrrr numpyrZlangchain_core.documentsr Zlangchain_core.embeddingsr Zlangchain_core.utilsr Zlangchain_core.vectorstoresrZ&langchain_community.vectorstores.utilsrrrr r!rrrrs $