o Zh'8@sddlZddlZddlmZmZmZmZmZmZddl Z ddl m Z m Z ddl mZddlmZddlmZeeZGdddeZdS) N)Any AsyncIteratorDictIteratorListOptional)AsyncCallbackManagerForLLMRunCallbackManagerForLLMRun)LLM)GenerationChunk)Fieldc @seZdZUdZeed< dZeeed< dZee ed< e ddd Z e ed< d Z eeed < d Zeeed < dZeeed< dZeeed< dZeeed< dZeeed< dZeeed< dZee ed< dZee ed< dZee ed< dZeeed< dZeeed< e ddd Ze ed< e ddd Ze ed< e dd d Ze ed < d!Zee ed"< e dd#d Ze ed#< e dd$d Ze ed$< gZ ee!eed%< dZ"e ed&< e#d'e$ee%ffd(d)Z&e#d'e$ee%ffd*d+Z'e#d'efd,d-Z(dTextGenazText generation models from WebUI. To use, you should have the text-generation-webui installed, a model loaded, and --api added as a command-line option. Suggested installation, use one-click installer for your OS: https://github.com/oobabooga/text-generation-webui#one-click-installers Parameters below taken from text-generation-webui api example: https://github.com/oobabooga/text-generation-webui/blob/main/api-examples/api-example.py Example: .. code-block:: python from langchain_community.llms import TextGen llm = TextGen(model_url="http://localhost:8500") model_urlNpresetmax_new_tokensT do_sample)aliasg? temperatureg?top_p typical_prepsilon_cutoff eta_cutoffgzG?repetition_penalty(top_k min_lengthno_repeat_ngram_size num_beams penalty_alphalength_penaltyFearly_stoppingseed add_bos_tokenitruncation_length ban_eos_tokenskip_special_tokensstopping_strings streamingreturncCsid|jd|jd|jd|jd|jd|jd|jd|jd |jd |j d |j d |j d |j d|j d|jd|jd|j|j|j|j|jdS)z/Get the default parameters for calling textgen.rrrrrrrrrrrrr r!r"r$r%)r&r'r(r))rrrrrrrrrrrrr r!r"r$r%r&r'r(r)selfr.W/var/www/html/lang_env/lib/python3.10/site-packages/langchain_community/llms/textgen.py_default_paramsvsP     zTextGen._default_paramscCsid|ji|jS)zGet the identifying parameters.r)rr0r,r.r.r/_identifying_paramsszTextGen._identifying_paramscCsdS)zReturn type of llm.Ztextgenr.r,r.r.r/ _llm_typeszTextGen._llm_typestopcCsH|jr |dur td|jdur|j}nd|ji}|jp|pg|d<|S)a Performs sanity check, preparing parameters in format needed by textgen. Args: stop (Optional[List[str]]): List of stop sequences for textgen. Returns: Dictionary containing the combined parameters. Nz2`stop` found in both the input and default params.rr))r) ValueErrorrr0)r-r3paramsr.r.r/_get_parameterss   zTextGen._get_parametersprompt run_managerkwargsc Ks|jrd}|jd |||d|D]}||j7}q|}|S|jd}||} | } || d<tj|| d} | jdkrH| ddd }|St d | d}|S) Call the textgen web API and return the output. Args: prompt: The prompt to use for generation. stop: A list of strings to stop generation when encountered. Returns: The generated text. Example: .. code-block:: python from langchain_community.llms import TextGen llm = TextGen(model_url="http://localhost:5000") llm.invoke("Write a story about llamas.") r7r3r8/api/v1/generater7jsonresultsrtextERROR: response: Nr.) r*_streamrBrr6copyrequestspost status_coder?print r-r7r3r8r9Zcombined_text_outputchunkresulturlr5requestresponser.r.r/_calls*   z TextGen._callc s|jr#d}|jd |||d|2z 3dHW}||j7}q6|}|S|jd}||} | } || d<tj|| d} | jdkrN| dd d }|St d | d}|S) r:r;r<Nr=r7r>r@rArrBrCr.) r*_astreamrBrr6rErFrGrHr?rIrJr.r.r/_acalls.   zTextGen._acallc kszddl}Wn tytdwi|||}|jd}|}||d<|} | || t | | } t | } | ddkr`t | d dd } |r\|j | jd | Vn | dd krl| dSq< aYields results objects as they are generated in real time. It also calls the callback manager's on_llm_new_token event with similar parameters to the OpenAI LLM class method of the same name. Args: prompt: The prompts to pass into the model. stop: Optional list of stop words to use when generating. Returns: A generator representing the stream of tokens being generated. Yields: A dictionary like objects containing a string token and metadata. See text-generation-webui docs and below for more. Example: .. code-block:: python from langchain_community.llms import TextGen llm = TextGen( model_url = "ws://localhost:5005" streaming=True ) for chunk in llm.stream("Ask 'Hi, how are you?' like a pirate:'", stop=["'"," "]): print(chunk, end='', flush=True) # noqa: T201 rNz9The `websocket-client` package is required for streaming.z/api/v1/streamr7TeventZ text_streamrB)rBZgeneration_info)tokenZ stream_end websocket ImportErrorr6rrEZ WebSocketconnectsendr?dumpsrecvloadsr Zon_llm_new_tokenrBclose r-r7r3r8r9rWr5rMrNZwebsocket_clientrLrKr.r.r/rDs<$       zTextGen._streamc Kszddl}Wn tytdwi|||}|jd}|}||d<|} | || t | | } t | } | ddkrct | d dd } |r_|j | jd IdH| Vn | dd kro| dSqs