o Zh[z@sddlZddlmZmZmZmZmZmZmZddl Z ddl m Z ddl m Z ddlmZddlmZmZddlmZmZddl mZGd d d e ZGd d d e ZdS) N)AnyDictIteratorListOptionalTupleUnion)CallbackManagerForLLMRun)LLM)GenerationChunk)convert_to_secret_strget_from_dict_or_env)Field SecretStr)Responsec s2eZdZUdZeddZeed< eeddZ eed< edddZ eed< edddZ eed < ed dZ e ed < d Zeeeefed < GdddZede fddZedeeeffddZedeeeffddZedefddZdedd ffdd ZdedeeeffddZd/deeedeeeffd d!Z d0d"eeeefdeeed ee defd#d$Z d%edefd&d'Z!d%ede"e#fd(d)Z$ d1d"eeeefdeeed*ee%dede"e#f d+d,Z& d1d"eeeefdeeed*ee%dedef d-d.Z'Z(S)2 SambaStudiou SambaStudio large language models. Setup: To use, you should have the environment variables ``SAMBASTUDIO_URL`` set with your SambaStudio environment URL. ``SAMBASTUDIO_API_KEY`` set with your SambaStudio endpoint API key. https://sambanova.ai/products/enterprise-ai-platform-sambanova-suite read extra documentation in https://docs.sambanova.ai/sambastudio/latest/index.html Example: .. code-block:: python from langchain_community.llms.sambanova import SambaStudio SambaStudio( sambastudio_url="your-SambaStudio-environment-URL", sambastudio_api_key="your-SambaStudio-API-key, model_kwargs={ "model" : model or expert name (set for Bundle endpoints), "max_tokens" : max number of tokens to generate, "temperature" : model temperature, "top_p" : model top p, "top_k" : model top k, "do_sample" : wether to do sample "process_prompt": wether to process prompt (set for Bundle generic v1 and v2 endpoints) }, ) Key init args — completion params: model: str The name of the model to use, e.g., Meta-Llama-3-70B-Instruct-4096 (set for Bundle endpoints). streaming: bool Whether to use streaming handler when using non streaming methods model_kwargs: dict Extra Key word arguments to pass to the model: max_tokens: int max tokens to generate temperature: float model temperature top_p: float model top p top_k: int model top k do_sample: bool wether to do sample process_prompt: wether to process prompt (set for Bundle generic v1 and v2 endpoints) Key init args — client params: sambastudio_url: str SambaStudio endpoint Url sambastudio_api_key: str SambaStudio endpoint api key Instantiate: .. code-block:: python from langchain_community.llms import SambaStudio llm = SambaStudio=( sambastudio_url = set with your SambaStudio deployed endpoint URL, sambastudio_api_key = set with your SambaStudio deployed endpoint Key, model_kwargs = { "model" : model or expert name (set for Bundle endpoints), "max_tokens" : max number of tokens to generate, "temperature" : model temperature, "top_p" : model top p, "top_k" : model top k, "do_sample" : wether to do sample "process_prompt" : wether to process prompt (set for Bundle generic v1 and v2 endpoints) } ) Invoke: .. code-block:: python prompt = "tell me a joke" response = llm.invoke(prompt) Stream: .. code-block:: python for chunk in llm.stream(prompt): print(chunk, end="", flush=True) Async: .. code-block:: python response = llm.ainvoke(prompt) await response defaultsambastudio_urlsambastudio_api_keyT)rexcludebase_url streaming_urlF streamingN model_kwargsc@eZdZdZdS)zSambaStudio.ConfigTN__name__ __module__ __qualname__Zpopulate_by_namer!r!Y/var/www/html/lang_env/lib/python3.10/site-packages/langchain_community/llms/sambanova.pyConfig|r#returncCdS)9Return whether this model can be serialized by Langchain.Tr!clsr!r!r"is_lc_serializablezSambaStudio.is_lc_serializablecCs dddS)Nrr)rrr!selfr!r!r" lc_secretsszSambaStudio.lc_secretscCsd|jid|jiS)Return a dictionary of identifying parameters. This information is used by the LangChain callback system, which is used for tracing purposes make it possible to monitor LLMs. rr)rrr,r!r!r"_identifying_paramsszSambaStudio._identifying_paramscCr&)zReturn type of llm.zsambastudio-llmr!r,r!r!r" _llm_typer+zSambaStudio._llm_typekwargsc sTt|dd|d<tt|dd|d<||d\|d<|d<tjdi|dS) 'init and validate environment variablesrZSAMBASTUDIO_URLrZSAMBASTUDIO_API_KEYrrNr!)r r _get_sambastudio_urlssuper__init__r-r2 __class__r!r"r6s zSambaStudio.__init__urlcCsdd|vr |}|}||fSd|vr|dd}|}||fS|}d|vr.d|d}||fStd)a3 Get streaming and non streaming URLs from the given URL Args: url: string with sambastudio base or streaming endpoint url Returns: base_url: string with url to do non streaming calls streaming_url: string with url to do streaming calls chat/completionsstreamzstream/rZgenericzgeneric/streamUnsupported URL)replacejoinsplit ValueError)r-r:rZ stream_urlr!r!r"r4s   z!SambaStudio._get_sambastudio_urlsstopcCsB|durg}|jp i}|dg|}t|dkr||d<d|jvrKd|vr/|d|d<d|vr<|d|d<d |vrG|d |}|Sd |jvrnd|vr]|d|d<d|vrj|d|d<|}|Sd |jvrd|vr|d|d<d|vr|d|d<d d |D}|Std|jd)aD Get the tuning parameters to use when calling the LLM. Args: stop: Stop words to use when generating. Model output is cut off at the first occurrence of any of the stop substrings. Returns: The tuning parameters in the format required by api to use NZstop_sequencesrr;Z select_expertmodelZmax_tokens_to_generate max_tokensprocess_promptapi/v2/predict/genericapi/predict/genericcSs&i|]\}}|t|jt|dqS))typevalue)rHrstr).0kvr!r!r" sz2SambaStudio._get_tuning_params..r=9only openai, generic v1 and generic v2 APIs are supported)rgetlenrkeyspopitemsrA)r-rBZ _model_kwargsZ_stop_sequencesZ tuning_paramsr!r!r"_get_tuning_paramssP              zSambaStudio._get_tuning_paramspromptc Cst|tr|g}||}d|jvr6d|ddg}||d|}dd|D}d|jd d }nd |jvrq|d d rQt ddd|ddgd}n|d}d|dg}dd|D}||d}d|ji}nLd|jvr|d d r|d ddkrt ddd|ddgd}n |d}n|d}|r||d}n|g|d}d|ji}n t d|jdt } |r| j |j||dd} n | j |j||d d} | jd krtd!| jd"| jd"| S)#a Performs a post request to the LLM API. Args: prompt: The prompt to pass into the model stop: list of stop tokens streaming: wether to do a streaming call Returns: A request Response object r;userrrolecontent)messagesr<cSi|] \}}|dur||qSNr!rKkeyrIr!r!r"rNz/SambaStudio._handle_request..Bearer application/json Authorizationz Content-TyperFrEFzsambaverse-conversation-idN)Z message_idrYrZ)Zconversation_idr[Zitem0)idrIcSr\r]r!r^r!r!r"rN2r`)rTparamsr_rGrITrue)instancerf)Z instancesrfr=rOTheadersjsonr<2Sambanova / complete call failed with status code .) isinstancerJrUrrTrget_secret_valuerPrkdumpsrArequestsSessionpostrr status_code RuntimeErrortext) r-rVrBrrf messages_dictdatarjrT http_sessionresponser!r!r"_handle_requests~                zSambaStudio._handle_requestr{c Csz|}Wnty}z td|d|jd}~wwd|jvr/|dddd}|Sd |jvr@|d dd d }|Sd |jvrO|ddd }|Std|jd) Process a non streaming response from the api Args: response: A request Response object Returns completion: a string with model generation ;Sambanova /complete call failed couldn't get JSON response response: Nr;choicesrmessagerZrFrTrI completionrGZ predictionsr=rO)rk ExceptionrvrwrrAr-r{Z response_dicterr!r!r"_process_responsehs,     zSambaStudio._process_responsec csLzddl}Wn tytdwd|jvr||}|D]{}|jdkr5td|jd|jdzP|jdkrt |jt rHt |j}n td|jd|jd| d rgtd|jd|jdt|d dkrz|d dd d }nd }t|d}|VWq!ty}z td|d|jd}~wwdSd|jvr|D]3} zt | }|ddddd}t|d}|VWqty}z td|d| d}~wwdSd|jvr|D]2} zt | }|dddd}t|d}|VWqty}z td|d| d}~wwdStd|jd) Process a streaming response from the api Args: response: An iterable request Response object Yields: GenerationChunk: a GenerationChunk with model partial generation rNTcould not import sseclient libraryPlease install it with `pip install sseclient-py`.r; error_event1Sambanova /complete call failed with status code rn[DONE]errorrdeltarZrrw3Error getting content chunk raw streamed response: data: rFresultrTrIZ stream_tokenzline: rG responsesr=rO) sseclient ImportErrorr SSEClienteventseventrvruryrorJrkloadsrPrQr r iter_linesrA) r-r{rclientrryrZgenerated_chunkrliner!r!r"_process_stream_responses           &          z$SambaStudio._process_stream_response run_managerck<|j||dd}||D] }|r||j|VqdS)a\Call out to Sambanova's complete endpoint. Args: prompt: The prompt to pass into the model. stop: a list of strings on which the model should stop generating. run_manager: A run manager with callbacks for the LLM. Yields: chunk: GenerationChunk with model partial generation TrNr|rZon_llm_new_tokenrwr-rVrBrr2r{chunkr!r!r"_stream zSambaStudio._streamcKT|jrd}|jd|||d|D]}||j7}q|S|j||dd}||}|S)a Call out to Sambanova's complete endpoint. Args: prompt: The prompt to pass into the model. stop: a list of strings on which the model should stop generating. Returns: result: string with model generation rrVrBrFrNr!rrrwr|rr-rVrBrr2rrr{r!r!r"_call  zSambaStudio._callr]NFNN))rrr __doc__rrrJ__annotations__rrrrrboolrrrrr# classmethodr*propertyr.r0r1r6rr4rrUrrr|rrr rr rr __classcell__r!r!r8r"r s \$D  f$b   rc s2eZdZUdZeddZeed< eeddZ eed< eddZ eed< eddZ e ed < ed dZ eed < ed dZeed < eddZeeed< eddZeeed< eddidZeed< GdddZede fddZedeeeffddZedeeeffddZedefddZdeddffd d! Z  d0d"e e!eefd#ee!ed ee de"fd$d%Z#d&e"defd'd(Z$d&e"de%e&fd)d*Z'  d1d"e e!eefd#ee!ed+ee(dedef d,d-Z)  d1d"e e!eefd#ee!ed+ee(dede%e&f d.d/Z*Z+S)2SambaNovaCloudu SambaNova Cloud large language models. Setup: To use, you should have the environment variables: ``SAMBANOVA_URL`` set with SambaNova Cloud URL. defaults to http://cloud.sambanova.ai/ ``SAMBANOVA_API_KEY`` set with your SambaNova Cloud API Key. Example: .. code-block:: python from langchain_community.llms.sambanova import SambaNovaCloud SambaNovaCloud( sambanova_api_key="your-SambaNovaCloud-API-key, model = model name, max_tokens = max number of tokens to generate, temperature = model temperature, top_p = model top p, top_k = model top k ) Key init args — completion params: model: str The name of the model to use, e.g., Meta-Llama-3-70B-Instruct-4096 (set for CoE endpoints). streaming: bool Whether to use streaming handler when using non streaming methods max_tokens: int max tokens to generate temperature: float model temperature top_p: float model top p top_k: int model top k Key init args — client params: sambanova_url: str SambaNovaCloud Url defaults to http://cloud.sambanova.ai/ sambanova_api_key: str SambaNovaCloud api key Instantiate: .. code-block:: python from langchain_community.llms.sambanova import SambaNovaCloud SambaNovaCloud( sambanova_api_key="your-SambaNovaCloud-API-key, model = model name, max_tokens = max number of tokens to generate, temperature = model temperature, top_p = model top p, top_k = model top k ) Invoke: .. code-block:: python prompt = "tell me a joke" response = llm.invoke(prompt) Stream: .. code-block:: python for chunk in llm.stream(prompt): print(chunk, end="", flush=True) Async: .. code-block:: python response = llm.ainvoke(prompt) await response rr sambanova_urlsambanova_api_keyzMeta-Llama-3.1-8B-InstructrCFrirDgffffff? temperatureNtop_ptop_kZ include_usageTstream_optionsc@r)zSambaNovaCloud.ConfigTNrr!r!r!r"r#{r$r#r%cCr&)r'Fr!r(r!r!r"r*~r+z!SambaNovaCloud.is_lc_serializablecCsddiS)Nrr!r,r!r!r"r.szSambaNovaCloud.lc_secretscCs"|j|j|j|j|j|j|jdS)r/rCrrDrrrrrr,r!r!r"r0sz"SambaNovaCloud._identifying_paramscCr&)z7Get the type of language model used by this chat model.zsambanovacloud-llmr!r,r!r!r"r1r+zSambaNovaCloud._llm_typer2c s>t|dddd|d<tt|dd|d<tjdi|dS) r3rZ SAMBANOVA_URLz,https://api.sambanova.ai/v1/chat/completionsrrZSAMBANOVA_API_KEYNr!)r r r5r6r7r8r!r"r6s  zSambaNovaCloud.__init__rVrBc Cst|tr|g}d|ddg}|||j||j|j|j|jd}dd|D}d|j dd }t }|rE|j |j ||d d }n |j |j ||d d }|jd kratd|jd|jd|S)z Performs a post request to the LLM API. Args: prompt: The prompt to pass into the model. stop: list of stop tokens Returns: A request Response object rWrrX)r[r<rDrBrCrrrcSr\r]r!r^r!r!r"rNr`z2SambaNovaCloud._handle_request..rarbrcTriFrlrmrn)rorJrDrCrrrrTrrprrrsrtrrurvrw) r-rVrBrrxryrjrzr{r!r!r"r|sB     zSambaNovaCloud._handle_requestr{c CsTz|}Wnty}z td|d|jd}~ww|dddd}|S)r}r~rNrrrrZ)rkrrvrwrr!r!r"rs z SambaNovaCloud._process_responsec cs4zddl}Wn tytdw||}|D]{}|jdkr0td|jd|jdzP|jdkrt|jt rCt |j}n td|jd|jd| drbtd|jd|jdt |d dkru|d dd d }nd }t|d }|VWqty}z td|d|jd}~wwdS)rrNrrrrnrrrrrZrrrr)rrrrrrvruryrorJrkrrPrQr r) r-r{rrrryrZrrr!r!r"rsh         z'SambaNovaCloud._process_stream_responsercKr)Call out to SambaNovaCloud complete endpoint. Args: prompt: The prompt to pass into the model. stop: Optional list of stop words to use when generating. Returns: The string generated by the model. rrFrNr!rrr!r!r"r1rzSambaNovaCloud._callckr)rTrNrrr!r!r"rNrzSambaNovaCloud._streamrr),rrr rrrrJrrrrCrrrDintrfloatrrrrdictr#rr*rrr.rr0r1r6rrrr|rrr rr rrrr!r!r8r"rs @  6<    r)rktypingrrrrrrrrrZ langchain_core.callbacks.managerr Z#langchain_core.language_models.llmsr Zlangchain_core.outputsr Zlangchain_core.utilsr r Zpydanticrrrrrr!r!r!r"s$