o Zh. @sdZddlmZmZmZmZmZmZddlm Z m Z ddl m Z m Z mZddlmZmZmZddlmZmZddlmZmZmZddlmZd Zd Zd eed eeefd eeefddfddZdedeed eee fdedef ddZ!GdddeZ"dS)z Wrapper around Anyscale Endpoint)AnyDictListMappingOptionalSet)AsyncCallbackManagerForLLMRunCallbackManagerForLLMRun) GenerationGenerationChunk LLMResult)convert_to_secret_strget_from_dict_or_envpre_init)Field SecretStr) BaseOpenAIacompletion_with_retrycompletion_with_retry) is_openai_v1z%https://api.endpoints.anyscale.com/v1z$mistralai/Mixtral-8x7B-Instruct-v0.1keysresponse token_usagereturnNcCsN||d}|D]}||vr|d|||<q |||d|7<q dS)zUpdate token usage.usageN) intersection)rrrZ _keys_to_use_keyrX/var/www/html/lang_env/lib/python3.10/site-packages/langchain_community/llms/anyscale.pyupdate_token_usages rchoicesprompts model_namec Csdg}t|D] \}}||}|t|ddt|d|dddgq||d}t||dS) z2Create the LLMResult from the choices and prompts.messagecontent finish_reasonlogprobs)r%r&)textgeneration_info)rr") generations llm_output) enumerateappendr dictgetr ) r r!rr"r)i_choicer*rrrcreate_llm_result+s  r2c sBeZdZUdZ eedZeed<ee ddZ e ed<ee dZ eed<ee dZeed<ed efd d Zed ed efd dZed eeefffdd Zed eeefffdd Zed efddZ  ddeedeeedeeded ef ddZ  ddeedeeedee ded ef ddZ!Z"S)AnyscaleaAnyscale large language models. To use, you should have the environment variable ``ANYSCALE_API_KEY``set with your Anyscale Endpoint, or pass it as a named parameter to the constructor. To use with Anyscale Private Endpoint, please also set ``ANYSCALE_BASE_URL``. Example: .. code-block:: python from langchain.llms import Anyscale anyscalellm = Anyscale(anyscale_api_key="ANYSCALE_API_KEY") # To leverage Ray for parallel processing @ray.remote(num_cpus=1) def send_query(llm, text): resp = llm.invoke(text) return resp futures = [send_query.remote(anyscalellm, text) for text in texts] results = ray.get(futures) defaultanyscale_api_baseanyscale_api_keyr")default_factoryprefix_messagesrcCdS)NFr)clsrrris_lc_serializable\szAnyscale.is_lc_serializablevaluescCs t|ddtd|d<tt|dd|d<t|ddtd|d<zHdd l}trR|d|dd }|d sA|jdi|j |d <|d sQ|j di|j |d <n|d|d <|d|d<|j |d <Wn t yqt dw|dr|ddkrt d|dr|ddkrt d|S)z?Validate that api key and python package exists in environment.r6ZANYSCALE_API_BASEr4r8ZANYSCALE_API_KEYr"Z MODEL_NAMErN)api_keybase_urlclientZ async_clientZopenai_api_baseZopenai_api_keyzTCould not import openai python package. Please install it with `pip install openai`. streamingnz!Cannot stream results when n > 1.Zbest_ofz'Cannot stream results when best_of > 1.r)rDEFAULT_BASE_URLr DEFAULT_MODELopenairget_secret_valuer.ZOpenAIZ completionsZ AsyncOpenAIZ Completion ImportError ValueError)r<r>rGZ client_paramsrrrvalidate_environment`sX         zAnyscale.validate_environmentcsid|jitjS)zGet the identifying parameters.r")r"super_identifying_paramsself __class__rrrMs zAnyscale._identifying_paramscs8d|ji}ts||j|jdi|tjS)z,Get the parameters used to invoke the model.model)r?Zapi_base)r"rupdater8rHr6rL_invocation_params)rOZ openai_credsrPrrrTszAnyscale._invocation_paramscCr;)zReturn type of llm.z Anyscale LLMrrNrrr _llm_typeszAnyscale._llm_typeNr!stop run_managerkwargscKs@|j}i||}||||}g}i}hd} d} |D]y} |jrkt| dkr+tdd} |j| d||fi|D] } | durC| } q:| | 7} q:| dusNJ|| j| jr[| j dnd| jre| j dnddqt |f| d|d |}t |t s| }| |d t| ||| s| d } q|j||||| d S) atCall out to OpenAI's endpoint with k unique prompts. Args: prompts: The prompts to pass into the model. stop: Optional list of stop words to use when generating. Returns: The full LLM output. Example: .. code-block:: python response = openai.generate(["Tell me a joke."]) Z total_tokensZcompletion_tokensZ prompt_tokensNrD,Cannot stream results with multiple prompts.rr%r&r'r%r&promptrWr system_fingerprintr^)rTget_sub_promptsrBlenrJ_streamr,r'r(r.r isinstancer-extendrr2rOr!rVrWrXparamsZ sub_promptsr rZ_keysr^Z_promptsZ generationchunkrrrr _generatesb        zAnyscale._generatecsD|j}i||}||||}g}i}hd} d} |D]z} |jrqt| dkr,tdd} |j| d||fi|2z3dHW} | durH| } q;| | 7} q;6| dusTJ|| j| jra| j dnd| jrk| j dnddqt |f| d|d |IdH}t |t s| }| |d t| ||q|j||||| d S) z:Call out to OpenAI's endpoint async with k unique prompts.rYNrDrZrr%r&r[r\r r_)rTr`rBrarJZ_astreamr,r'r(r.rrcr-rdrr2rerrr _ageneratesh       zAnyscale._agenerate)NN)#__name__ __module__ __qualname____doc__rrEr6str__annotations__rr8rFr"listr:r classmethodboolr=rrrKpropertyrrrMrTrUrr r rhrri __classcell__rrrPrr3AsV 7  R r3)#rmtypingrrrrrrZlangchain_core.callbacksrr Zlangchain_core.outputsr r r Zlangchain_core.utilsr rrZpydanticrrZlangchain_community.llms.openairrrZ langchain_community.utils.openairrErFrnrintr2r3rrrrs>