o Zhh@sddlmZddlZddlmZmZmZmZmZm Z m Z ddl m Z ddl mZddlmZer6ddlmZdd d d d ZeeZGd d d eZdS)) annotationsN) TYPE_CHECKINGAnyDictIterableListOptionalTuple)Document) Embeddings) VectorStore HippoClient localhostZ7788Zadmin)hostportusernamepasswordc@seZdZdZ       dNdOddZdPddZ dQdRd!d"Z dSdTd$d%ZdUd&d'ZdVd(d)Z dUd*d+Z   ,dWdXd5d6Z 7   dYdZd>d?Z 7   dYd[dAdBZ 7   dYd\dEdFZedddedddfd]dLdMZdS)^Hippoa`Hippo` vector store. You need to install `hippo-api` and run Hippo. Please visit our official website for how to run a Hippo instance: https://www.transwarp.cn/starwarp Args: embedding_function (Embeddings): Function used to embed the text. table_name (str): Which Hippo table to use. Defaults to "test". database_name (str): Which Hippo database to use. Defaults to "default". number_of_shards (int): The number of shards for the Hippo table.Defaults to 1. number_of_replicas (int): The number of replicas for the Hippo table.Defaults to 1. connection_args (Optional[dict[str, any]]): The connection args used for this class comes in the form of a dict. index_params (Optional[dict]): Which index params to use. Defaults to IVF_FLAT. drop_old (Optional[bool]): Whether to drop the current collection. Defaults to False. primary_field (str): Name of the primary key field. Defaults to "pk". text_field (str): Name of the text field. Defaults to "text". vector_field (str): Name of the vector field. Defaults to "vector". The connection args used for this class comes in the form of a dict, here are a few of the options: host (str): The host of Hippo instance. Default at "localhost". port (str/int): The port of Hippo instance. Default at 7788. user (str): Use which user to connect to Hippo instance. If user and password are provided, we will add related header in every RPC call. password (str): Required when user is provided. The password corresponding to the user. Example: .. code-block:: python from langchain_community.vectorstores import Hippo from langchain_community.embeddings import OpenAIEmbeddings embedding = OpenAIEmbeddings() # Connect to a hippo instance on localhost vector_store = Hippo.from_documents( docs, embedding=embeddings, table_name="langchain_test", connection_args=HIPPO_CONNECTION ) Raises: ValueError: If the hippo-api python package is not installed. testdefaultNFembedding_functionr table_namestr database_namenumber_of_shardsintnumber_of_replicasconnection_argsOptional[Dict[str, Any]] index_paramsOptional[dict]drop_oldOptional[bool]c Cs$||_||_||_||_||_||_d|_d|_d|_g|_ |dur$t }| ||_ d|_ z|j |j|jrB|rB|j |j|jWnty\} z td|jd| d} ~ wwz|j |j|jrq|j |j|j|_ Wnty} z td|jd| d} ~ ww|dS)NpktextZvectorz+An error occurred while deleting the table z: z*An error occurred while getting the table )rrembedding_funcrrr!_primary_field _text_field _vector_fieldfieldsDEFAULT_HIPPO_CONNECTION_create_connection_aliashccolZcheck_table_existsZ delete_table Exceptionloggingerror get_table_get_env) selfrrrrrrr!r#er7]/var/www/html/lang_env/lib/python3.10/site-packages/langchain_community/vectorstores/hippo.py__init__PsP   zHippo.__init__dictreturnrc s zddlm}Wnty}ztd|d}~ww|dd}|dd|dd}|d d}|dur[dur[d |vrP|d }d fd d |D}nt|d t}ntdzt d|d||g||dWSt y}zt d|d}~ww)z*Create the connection to the Hippo server.rr zQUnable to import transwarp_hipp_api, please install with `pip install hippo-api`.NrrrZshivar,csg|] }|dqS):r7).0hrr7r8 sz2Hippo._create_connection_alias..r=z/Missing standard address type for reuse attemptzcreate HippoClient[])rpwdzFailed to create new connection) transwarp_hippo_api.hippo_clientr ImportErrorgetsplitjoinr ValueErrorloggerinfor0r2) r5rrr6rrrhostsZ given_addressr7r@r8r-s8      zHippo._create_connection_alias embeddingsOptional[list] metadatasOptional[List[dict]]NonecCs<td|durtd|||||dS)Nzinit ...zcreate collection)rJrK_create_collection_extract_fields _create_index)r5rMrOr7r7r8r4s    zHippo._get_envlistc CsDddlm}ddlm}t|d}td|g}|||jd|j |||j d|j |||j d|j d|id|rp|d D]&\}}t|tret|} |||d|j d| idqI|||d|j qItd ||jj|jd||j|j|jd |j|j|j|_td |jd |jd dS)Nr) HippoField) HippoTypez[_create_collection] dim: TF dimension)Z type_paramsz[_create_collection] fields: )nameZauto_idr+rrrz$[_create_collection] : create table z in z successfully)rDrVtranswarp_hippo_api.hippo_typerWlenrJdebugappendr(STRINGr)r*Z FLOAT_VECTORitems isinstancerUr.Z create_tablerrrrr3r/rK) r5rMrOrVrWdimr+keyvalueZ value_dimr7r7r8rRsZ      zHippo._create_collectioncCsbddlm}t|j|r/|jj}td||D] }|j|j qtd|jdSdS)z,Grab the existing fields from the Collectionr HippoTablez[_extract_fields] schema:z04 [_extract_fields] fields:N) rDrer`r/schemarJr\r+r]rY)r5rerfxr7r7r8rSs  zHippo._extract_fieldscCsddlm}t|j|rJ|j|j|j|ji}|dd}|dur&dS|j|j|j|jdD]}t d||d|j krI|Sq4dS)z0Return the vector index information if it existsrrdembedding_indexesNz[_get_index] embedding_indexes column) rDrer`r/r.Zget_table_inforrrFrJr\r*)r5reZ table_inforhrgr7r7r8 _get_indexs(    zHippo._get_indexc Csddlm}ddlm}m}t|j|r|dur|dur|jdur]d|j |j dd|_|jj |j |jd|jd |jd |jd d t |j|jdt d dS|j |j|j|j|jd}|j|j|j |j d}||jd |jd <|jd dkr||jd |jd <|j |j |jd|jd |jd t |j|jddS|jd dks|jd dkr||jd |jd <|jj |j |jd|jd |jd |jd d|jdddt |j|jddS|jd dkrB||jd |jd <|jj |j |jd|jd |jd |jd d|jdd|jdd|jddt |j|jddS|jd dkr||jd |jd <|jj |j |jd|jd |jd |jd|jd|jddt |j|jddStddSdSdS) z Create a index on the collectionrrd) IndexType MetricTypeNZlangchain_auto_create ) index_name metric_type index_typenlistrnrprorq)rqzcreate index successfully)IVF_FLATFLATIVF_SQIVF_PQHNSW)ipIPl2L2rsrrrtnprobe)rqr{runbitsm)rqr{r|r~rvMef_construction ef_search)rrrzeIndex name does not match, please enter the correct index name. (FLAT, IVF_FLAT, IVF_PQ,IVF_SQ, HNSW))rDrerZrkrlr`r/rjr!rzrrZ create_indexr*rJr\Zactivate_indexrKrsrtrurvrxrFrI)r5rerkrlZ index_dictZ metric_dictr7r7r8rTs                   zHippo._create_indextexts Iterable[str]timeout Optional[int] batch_sizekwargsr List[str]c sddlm}|rtdd|DrtdgSt|}td|zj|}Wnty>fdd|D}Ynwt |dkrLtdgStd t |t j |sb ||j |j|itd |td j|d ur|D]}|D]\} } | jvr| g| qqtj j} t | } d jvrjd td| td| |D]:t|| fddjD} zj | }td|Wqty}z td| |d }~wwdgS)a Add text to the collection. Args: texts: An iterable that contains the text to be added. metadatas: An optional list of dictionaries, each dictionary contains the metadata associated with a text. timeout: Optional timeout, in seconds. batch_size: The number of texts inserted in each batch, defaults to 1000. **kwargs: Other optional parameters. Returns: A list of strings, containing the unique identifiers of the inserted texts. Note: If the collection has not yet been created, this method will create a new collection. rrdcss|]}|dkVqdS)Nr7)r>tr7r7r8 sz"Hippo.add_texts..zNothing to insert, skipping.z[add_texts] texts: csg|]}j|qSr7)r' embed_queryr>rg)r5r7r8rAz#Hippo.add_texts..zdd_texts] len_embeddings:z[add_texts] metadatas:z[add_texts] fields:Nr%z[add_texts] total_count:csg|] }|qSr7r7r)endi insert_dictr7r8rAsz05 [add_texts] insert z0Failed to insert batch starting at entity: %s/%sr)rDreallrJr\rUr'Zembed_documentsNotImplementedErrorr[r`r/r4r)r*r+r_ setdefaultr]removerangeminZ insert_rowsrKr0r2)r5rrOrrrrerMdrbrcvectorsZ total_countZ insert_listresr6r7)rrrr5r8 add_textssb            zHippo.add_textsquerykparamexpr Optional[str]List[Document]cKsB|jdur tdgS|jd|||||d|}dd|DS)a Perform a similarity search on the query string. Args: query (str): The text to search for. k (int, optional): The number of results to return. Default is 4. param (dict, optional): Specifies the search parameters for the index. Defaults to None. expr (str, optional): Filtering expression. Defaults to None. timeout (int, optional): Time to wait before a timeout error. Defaults to None. kwargs: Keyword arguments for Collection.search(). Returns: List[Document]: The document results of the search. N!No existing collection to search.)rrrrrcSsg|]\}}|qSr7r7)r>doc_r7r7r8rAsz+Hippo.similarity_search..r7)r/rJr\similarity_search_with_score)r5rrrrrrrr7r7r8similarity_searchs   zHippo.similarity_searchList[Tuple[Document, float]]c KsD|jdur tdgS|j|}|jd|||||d|}|S)a Performs a search on the query string and returns results with scores. Args: query (str): The text being searched. k (int, optional): The number of results to return. Default is 4. param (dict): Specifies the search parameters for the index. Default is None. expr (str, optional): Filtering expression. Default is None. timeout (int, optional): The waiting time before a timeout error. Default is None. kwargs: Keyword arguments for Collection.search(). Returns: List[float], List[Tuple[Document, any, any]]: Nr) embeddingrrrrr7)r/rJr\r'r&similarity_search_with_score_by_vector) r5rrrrrrrretr7r7r8rs    z"Hippo.similarity_search_with_scorer List[float]c sH|jdur tdgS|jdd}||jtd|jtd|gtd|td|td||jj|j|g|||dtd |jd }g} d } tfd d |DD]5} ddt|| D} t | |j| d} tdd |d || }| d7} | | |fql| S)a Performs a search on the query string and returns results with scores. Args: embedding (List[float]): The embedding vector being searched. k (int, optional): The number of results to return. Default is 4. param (dict): Specifies the search parameters for the index. Default is None. expr (str, optional): Filtering expression. Default is None. timeout (int, optional): The waiting time before a timeout error. Default is None. kwargs: Keyword arguments for Collection.search(). Returns: List[Tuple[Document, float]]: Resulting documents and scores. Nrz search_field:zvectors:zoutput_fields:ztopk:zdsl:)Z search_fieldr output_fieldsZtopkZdslz-[similarity_search_with_score_by_vector] res:z%scoresrcsg|]}d|qS)rr7)r>fieldrr7r8rAfrz@Hippo.similarity_search_with_score_by_vector..cSsi|]\}}||qSr7r7)r>rrcr7r7r8 gsz@Hippo.similarity_search_with_score_by_vector..)Z page_contentmetadataz;[similarity_search_with_score_by_vector] res[0][score_col]:r) r/rJr\r+rr*rr)zipr popr])r5rrrrrrrZ score_colrcountr_metarZscorer7rr8r.sB     z,Hippo.similarity_search_with_score_by_vectorDict[str, Any]Optional[Dict[Any, Any]] search_paramsbool'Hippo'c Ksd|duri}td|d|||||| d| } td|td|| j||d| S)a Creates an instance of the VST class from the given texts. Args: texts (List[str]): List of texts to be added. embedding (Embeddings): Embedding model for the texts. metadatas (List[dict], optional): List of metadata dictionaries for each text.Defaults to None. table_name (str): Name of the table. Defaults to "test". database_name (str): Name of the database. Defaults to "default". connection_args (dict[str, Any]): Connection parameters. Defaults to DEFAULT_HIPPO_CONNECTION. index_params (dict): Indexing parameters. Defaults to None. search_params (dict): Search parameters. Defaults to an empty dictionary. drop_old (bool): Whether to drop the old collection. Defaults to False. kwargs: Other arguments. Returns: Hippo: An instance of the VST class. Nz'00 [from_texts] init the class of Hippo)rrrrr!r#z[from_texts] texts:z[from_texts] metadatas:)rrOr7)rJrKr\r) clsrrrOrrrr!rr#rZ vector_dbr7r7r8 from_textsss"#  zHippo.from_texts)rrrrNNF)rr rrrrrrrrrr r!r"r#r$)rr:r;r)NN)rMrNrOrPr;rQ)N)rMrUrOrPr;rQ)r;rQ)r;r )NNr) rrrOrPrrrrrrr;r)rNNN)rrrrrr"rrrrrrr;r)rrrrrr"rrrrrrr;r)rrrrrr"rrrrrrr;r)rrrr rOrPrrrrrrr!rrr r#rrrr;r)__name__ __module__ __qualname____doc__r9r-r4rRrSrjrTrrrr classmethodr,rr7r7r7r8rs\: 9"  A  w [ % * Er) __future__rr1typingrrrrrrr Zlangchain_core.documentsr Zlangchain_core.embeddingsr Zlangchain_core.vectorstoresr rDrr, getLoggerrrJrr7r7r7r8s $