o ZhY@sddlmZddlZddlZddlmZddlmZddlm Z m Z m Z m Z m Z mZddlmZddlmZddlmZdd lmZmZeZdddZGdddeZGdddeZGdddeZdS)) annotationsN)sha1)Thread)AnyDictIterableListOptionalTuple)Document) Embeddings) VectorStore) BaseSettingsSettingsConfigDictsstrargsrreturnboolcGs|D] }||vr dSqdS)z Check if a string contains multiple substrings. Args: s: string to check. *args: substrings to check. Returns: True if all substrings are in the string, False otherwise. FT)rrarr_/var/www/html/lang_env/lib/python3.10/site-packages/langchain_community/vectorstores/myscale.pyhas_mul_sub_strs  rc@seZdZUdZdZded<dZded<dZd ed <dZd ed <d Z ded <dZ ded<dddddZ ded<dZ ded<dZ ded<dZded<d'd d!Zed"d#d$d%d&ZdS)(MyScaleSettingsaMyScale client configuration. Attribute: myscale_host (str) : An URL to connect to MyScale backend. Defaults to 'localhost'. myscale_port (int) : URL port to connect with HTTP. Defaults to 8443. username (str) : Username to login. Defaults to None. password (str) : Password to login. Defaults to None. index_type (str): index type string. index_param (dict): index build parameter. database (str) : Database name to find the table. Defaults to 'default'. table (str) : Table name to operate on. Defaults to 'vector_table'. metric (str) : Metric to compute distance, supported are ('L2', 'Cosine', 'IP'). Defaults to 'Cosine'. column_map (Dict) : Column type map to project column name onto langchain semantics. Must have keys: `text`, `id`, `vector`, must be same size to number of columns. For example: .. code-block:: python { 'id': 'text_id', 'vector': 'text_embedding', 'text': 'text_plain', 'metadata': 'metadata_dictionary_in_json', } Defaults to identity map. localhostrhosti intportN Optional[str]usernamepasswordZMSTG index_typezOptional[Dict[str, str]] index_paramidtextvectormetadata)r#r$r%r&zDict[str, str] column_mapdefaultdatabaseZ langchaintableZCosinemetricitemrrcCs t||SN)getattr)selfr,rrr __getitem__Us zMyScaleSettings.__getitem__z.envutf-8Zmyscale_ignore)Zenv_fileZenv_file_encodingZ env_prefixextra)r,rrr)__name__ __module__ __qualname____doc__r__annotations__rrr r!r"r'r)r*r+r0rZ model_configrrrrr!s.            rcseZdZdZ dFdGfd d ZedHd dZdIddZdJddZdKddZ   dLdMd$d%Z e    dNdOd(d)Z dPd*d+Z dFdQd1d2Z dRdSd7d8Z 3 dRdTd9d:Z dRdUdd?Z  dWdXdBdCZedPdDdEZZS)YMyScalea`MyScale` vector store. You need a `clickhouse-connect` python package, and a valid account to connect to MyScale. MyScale can not only search with simple vector indexes. It also supports a complex query with multiple conditions, constraints and even sub-queries. For more information, please visit [myscale official site](https://docs.myscale.com/en/overview/) N embeddingr configOptional[MyScaleSettings]kwargsrrNonec  szddlm}Wn tytdwz ddlm}||_Wnty-dd|_Ynwt|dur;||_nt|_|jsDJ|jj rL|jj sNJ|jj r^|jj r^|jj r^|jjs`JdD] }||jj vslJqb|jjd vswJ|jjd vrtd t|d }|jjrd ddd|jjDnd}d|jj d|jj d|jj dd|jj dd|jj dd|jj dd|jj dd|d|jj dd|jjd|jjd |d!|jj dd"} ||_d#|_d$|_||_|jjd%vrd&nd'|_|d-|jj |jj |jj|jjd(||_z|j d)Wnt!yB} zt"d*|jj#d+WYd} ~ nd} ~ ww|j d,|j | dS).zMyScale Wrapper to LangChain embedding (Embeddings): config (MyScaleSettings): Configuration to MyScale Client Other keyword arguments will pass into [clickhouse-connect](https://docs.myscale.com/) r) get_clientzlCould not import clickhouse connect python package. Please install it with `pip install clickhouse-connect`.)tqdmcSs|Sr-r)xrrrsz"MyScale.__init__..N)r#r%r$r&)ZIPCOSINEL2)ipZcosinel2z_Lower case metric types will be deprecated the future. Please use one of ('IP', 'Cosine', 'L2')z try this out, ,cSs"g|] \}}d|d|dqS)'=r).0kvrrr "z$MyScale.__init__..z( CREATE TABLE IF NOT EXISTS .z( r#z String, r$r%z! Array(Float32), r&zP JSON, CONSTRAINT cons_vec_len CHECK length( z) = z$, VECTOR INDEX vidx z TYPE z&( 'metric_type=rIz,) ) ENGINE = MergeTree ORDER BY z \)rRrI)rCrDASCZDESC)rrrr z"SET allow_experimental_json_type=1zClickhouse version=z6 - There is no allow_experimental_json_type parameter.z$SET allow_experimental_object_type=1r)$Zclickhouse_connectr? ImportErrorr@pgbarsuper__init__r;rrrr'r)r*r+upperloggerwarninglen embed_queryr"joinitemsr!dimBS must_escape _embeddings dist_orderrr clientcommand ExceptiondebugZserver_version) r/r:r;r=r?r@rLr_Z index_paramsZschema__ __class__rrrWns        "            zMyScale.__init__cCs|jSr-)rbr/rrr embeddingsszMyScale.embeddingsvaluercsdfdd|DS)NrPc3s,|]}|jvrj|n|VqdSr-)rar`)rKcrkrr s*z%MyScale.escape_str..)r])r/rmrrkr escape_strszMyScale.escape_strtransacr column_names Iterable[str]c srd|}g}|D]}dfdd|D}|d|dq djjdjjd|dd|d }|S) NrHcs"g|] }dt|dqSrI)rpr)rKZ_nrkrrrNrOz'MyScale._build_istr..()z8 INSERT INTO TABLE rQz)) VALUES z )r]appendr;r)r*)r/rqrrks_datanZi_strrrkr _build_istrs zMyScale._build_istrcCs|||}|j|dSr-)r{rdre)r/rqrrZ_i_strrrr_inserts zMyScale._insert texts metadatasOptional[List[dict]] batch_sizeridsOptional[Iterable[str]] List[str]c Ks|pdd|D}|jj}g}|d||d||dt|jj|i}|p*dd|D}ttj|||d<tt|t|dksCJt | \} } z]d } |j t | d t|d D]4} t| | |jjd|j ksnJ|| t||kr| r| t|j|| gd } | g}qZt|dkr| r| ||| d d|DWSty} ztdt| dt| dgWYd } ~ Sd } ~ ww)aRun more texts through the embeddings and add to the vectorstore. Args: texts: Iterable of strings to add to the vectorstore. ids: Optional list of ids to associate with the texts. batch_size: Batch size of insertion metadata: Optional column data to be inserted Returns: List of ids from adding the texts into the vectorstore. cSsg|] }t|dqS)r1)rencode hexdigest)rKtrrrrNsz%MyScale.add_texts..r#r$r%cSsg|]}iqSrr)rKrhrrrrNr&rNzInserting data...)Zdesctotal)targetrcSsg|]}|qSrr)rKirrrrNr   )r;r'maprbr\jsondumpsr[setzipr^rUindexr_rwr]rr|startrfrYerrortyper)r/r~rrrr=Zcolmap_rqrrkeysvaluesrrMerrr add_textssF $     zMyScale.add_textsOptional[List[Dict[Any, Any]]]text_idsc Ks(|||fi|}|j||||d|S)aZCreate Myscale wrapper with existing texts Args: texts (Iterable[str]): List or tuple of strings to be added embedding (Embeddings): Function to extract text embedding config (MyScaleSettings, Optional): Myscale configuration text_ids (Optional[Iterable], optional): IDs for the texts. Defaults to None. batch_size (int, optional): Batchsize when transmitting data to MyScale. Defaults to 32. metadata (List[dict], optional): metadata to texts. Defaults to None. Other keyword arguments will pass into [clickhouse-connect](https://clickhouse.com/docs/en/integrations/python#clickhouse-connect-driver-api) Returns: MyScale Index )rrr)r) clsr~r:rr;rrr=ctxrrr from_textsszMyScale.from_textscCsd|jjd|jjd}||jjd|jjd7}|d|jjd7}|d7}|jd |jjd|jjD]}|d |d d d |dd d7}q;|d7}|S)zText representation for myscale, prints backends, username and schemas. Easy to use with `str(Myscale())` Returns: repr: string to show connection info and data schema z rQz @ :z zusername: z Table Schema: z4--------------------------------------------------- zDESC z|nameZ24sz |rz| ) r;r)r*rrrrdquery named_results)r/_reprrrrr__repr__>szMyScale.__repr__q_emb List[float]topk where_strrcCsdtt|}|rd|}nd}d|jjdd|jjdd|jjd |jjd |d |jjd d |d|jd|d }|S)NrH PREWHERE rP SELECT r$z, r&z, dist FROM rQ  ORDER BY distance(r%, []) AS dist  LIMIT )r]rrr;r'r)r*rcr/rrrZ q_emb_strq_strrrr _build_qstrR0     zMyScale._build_qstrrrLList[Document]cKs|j|j|||fi|S)aPerform a similarity search with MyScale Args: query (str): query string k (int, optional): Top K neighbors to retrieve. Defaults to 4. where_str (Optional[str], optional): where condition string. Defaults to None. NOTE: Please do not let end-user to fill this and always be aware of SQL injection. When dealing with metadatas, remember to use `{self.metadata_column}.attribute` instead of `attribute` alone. The default name for it is `metadata`. Returns: List[Document]: List of Documents )similarity_search_by_vectorrbr\)r/rrLrr=rrrsimilarity_searchfs zMyScale.similarity_searchc x|||}zfddj|DWSty;}ztdt|dt|dgWYd}~Sd}~ww)Perform a similarity search with MyScale by vectors Args: query (str): query string k (int, optional): Top K neighbors to retrieve. Defaults to 4. where_str (Optional[str], optional): where condition string. Defaults to None. NOTE: Please do not let end-user to fill this and always be aware of SQL injection. When dealing with metadatas, remember to use `{self.metadata_column}.attribute` instead of `attribute` alone. The default name for it is `metadata`. Returns: List[Document]: List of (Document, similarity) cs0g|]}t|jjd|jjddqS)r$r&Z page_contentr&r r;r'rKrrkrrrNsz7MyScale.similarity_search_by_vector..rrrN rrdrrrfrYrrrr/r:rLrr=rrrrkrr}  z#MyScale.similarity_search_by_vectorList[Tuple[Document, float]]c j|||}zfddj|DWSty?}ztdt |dt |dgWYd}~Sd}~ww)/Perform a similarity search with MyScale Args: query (str): query string k (int, optional): Top K neighbors to retrieve. Defaults to 4. where_str (Optional[str], optional): where condition string. Defaults to None. NOTE: Please do not let end-user to fill this and always be aware of SQL injection. When dealing with metadatas, remember to use `{self.metadata_column}.attribute` instead of `attribute` alone. The default name for it is `metadata`. Returns: List[Document]: List of documents most similar to the query text and cosine distance in float for each. Lower score represents more similarity. cs8g|]}t|jjd|jjdd|dfqS)r$r&rdistrrrkrrrNszCMyScale.similarity_search_with_relevance_scores..rrrN rrbr\rdrrrfrYrrrr/rrLrr=rrrrkr'similarity_search_with_relevance_scores  z/MyScale.similarity_search_with_relevance_scorescCs$|jd|jjd|jjdS)z, Helper function: Drop data zDROP TABLE IF EXISTS rQN)rdrer;r)r*rkrrrdropsz MyScale.dropOptional[List[str]]Optional[bool]c Ks|dur |dur Jdg}|r/t|dkr/ddd|D}||jjdd|d |r6||t|dks>Jd |}d |jjd |jjd |}z |j|WdSt yu}zt t |WYd}~dSd}~ww)a3Delete by vector ID or other criteria. Args: ids: List of ids to delete. **kwargs: Other keyword arguments that subclasses might use. Returns: Optional[bool]: True if deletion is successful, False otherwise, None if not implemented. NzIYou need to specify where to be deleted! Either with `ids` or `where_str`rrGcSsg|]}d|dqSrtr)rKr#rrrrNsz"MyScale.delete..r#z IN (rvz AND z DELETE FROM rQz WHERE TF) r[r]rwr;r'r)r*rdrerfrYrr) r/rrr=ZcondsZid_listZwhere_str_finalZqstrrrrrdeletes.   zMyScale.deletecCs |jjdS)Nr&)r;r'rkrrrmetadata_columns zMyScale.metadata_columnr-)r:r r;r<r=rrr>)rr )rmrrr)rqrrrrsrr)rqrrrrsrr>)Nr}N) r~rsrrrrrrr=rrr)NNNr})r~rsr:r rrr;r<rrrrr=rrr9rrrrrrrrrrrN) rrrLrrrr=rrr r:rrLrrrr=rrr rrrLrrrr=rrr)rr>)NN)rrrrr=rrr)r4r5r6r7rWpropertyrlrpr{r|r classmethodrrrrrrrrr __classcell__rrrirr9`sF`     8    % %  &r9csbeZdZdZdgfd#fd d Z d$d%ddZ  d&d'ddZ d&d(dd Zed)d!d"Z Z S)*MyScaleWithoutJSONzsMyScale vector store without metadata column This is super handy if you are working to a SQL-native table Nr:r r;r<must_have_colsrr=rrr>c s tj||fi|||_dS)agBuilding a myscale vector store without metadata column embedding (Embeddings): embedding model config (MyScaleSettings): Configuration to MyScale Client must_have_cols (List[str]): column names to be included in query Other keyword arguments will pass into [clickhouse-connect](https://docs.myscale.com/) N)rVrWr)r/r:r;rr=rirrrWs zMyScaleWithoutJSON.__init__rrrrrrrcCsdtt|}|rd|}nd}d|jjddd|jd|jjd|jjd |d |jjd d |d |jd|d }|S)NrHrrPrr$z, dist, z FROM rQrrr%rrr) r]rrr;r'rr)r*rcrrrrrrzMyScaleWithoutJSON._build_qstrrrLrc r)rcs4g|]tjjdfddjDdqS)r$ci|]}||qSrrrKrLrrr AzMMyScaleWithoutJSON.similarity_search_by_vector...rr r;r'rrKrkrrrN>szBMyScaleWithoutJSON.similarity_search_by_vector..rrrNrrrrkrr%rz.MyScaleWithoutJSON.similarity_search_by_vectorrrc r)rcs<g|]tjjdfddjDddfqS)r$crrrrrrrrdrzYMyScaleWithoutJSON.similarity_search_with_relevance_scores...rrrrrkrrrN`szNMyScaleWithoutJSON.similarity_search_with_relevance_scores..rrrNrrrrkrrIrz:MyScaleWithoutJSON.similarity_search_with_relevance_scorescCsdS)NrPrrkrrrrnsz"MyScaleWithoutJSON.metadata_column) r:r r;r<rrr=rrr>r-rrrrr) r4r5r6r7rWrrrrrrrrrirrs  % %r)rrrrrr) __future__rrlogginghashlibr threadingrtypingrrrrr r Zlangchain_core.documentsr Zlangchain_core.embeddingsr Zlangchain_core.vectorstoresr Zpydantic_settingsrr getLoggerrYrrr9rrrrrs$        ?