o ZhFQ@sZddlZddlZddlZddlmZddlmZddlm Z m Z m Z m Z m Z ddlmZddlmZddlmZmZddlmZmZdd lmZmZmZmZmZd ZGd d d eZ Gd dde!eZ"GdddZ#Gddde#Z$Gddde$Z%Gddde#Z&Gddde#Z'Gddde#Z(Gddde(Z)GdddeZ*Gdd d ee*Z+dS)!N)abstractmethod)Enum)AnyDictListMappingOptional)CallbackManagerForLLMRun)BaseLLM) Generation LLMResult)convert_to_secret_strget_from_dict_or_env) BaseModel ConfigDict SecretStrmodel_validator validator2c @sVeZdZdZdefdededededdf d d Z dd ed e e d e defddZ dS)AzureMLEndpointClientz AzureML Managed Endpoint client. endpoint_urlendpoint_api_keydeployment_nametimeoutreturnNcCs,|r|std||_||_||_||_dS)zInitialize the class.zXA key/token and REST endpoint should be provided to invoke the endpointN) ValueErrorrrrr)selfrrrrr`/var/www/html/lang_env/lib/python3.10/site-packages/langchain_community/llms/azureml_endpoint.py__init__s zAzureMLEndpointClient.__init__body run_managerkwargscKs\dd|jd}|jdkr|j|d<tj|j||}tjj||d|jd}| }|S)zcall.application/jsonzBearer )z Content-Type Authorizationrzazureml-model-deploymentr)r) rrurllibrequestRequestrurlopengetrread)rr!r"r#headersreqresponseresultrrrcall&s   zAzureMLEndpointClient.callN) __name__ __module__ __qualname____doc__DEFAULT_TIMEOUTstrintr bytesrr rr0rrrrrs2 rc@seZdZdZdZdZdZdS)AzureMLEndpointApiTypezAzure ML endpoints API types. Use `dedicated` for models deployed in hosted infrastructure (also known as Online Endpoints in Azure Machine Learning), or `serverless` for models deployed as a service with a pay-as-you-go billing or PTU. dedicatedrealtime serverlessN)r2r3r4r5r;r<r=rrrrr:@s r:c @seZdZUdZ dZeeed< dZeeed< dZ eed<e dedefd d Z e de efd d Zejfded ededefddZeejfdededefddZdS)ContentFormatterBasezZTransform request and response of AzureML endpoint to match with required schema. r$ content_typeacceptszError while formatting response payload for chat model of type `{api_type}`. Are you using the right formatter for the deployed model and endpoint type?format_error_msgpromptrcCs6dddddddd}|D] \}}|||}q|S) z*Escapes any special characters in `prompt`z\\z\"z\bz\fz\nz\rz\t)\"    )itemsreplace)rBZ escape_mapZescape_sequenceZescaped_sequencerrrescape_special_charactersys z.ContentFormatterBase.escape_special_characterscCtjgS)zSupported APIs for the given formatter. Azure ML supports deploying models using different hosting methods. Each method may have a different API structure.r:r;rrrrsupported_api_typessz(ContentFormatterBase.supported_api_types model_kwargsapi_typecCst)zFormats the request body according to the input schema of the model. Returns bytes or seekable file like object in the format specified in the content_type request header. )NotImplementedError)rrBrQrRrrrformat_request_payloads z+ContentFormatterBase.format_request_payloadoutputcCdS)zFormats the response body according to the output schema of the model. Returns the data type that is received from the response. Nr)rrUrRrrrformat_response_payloadsz,ContentFormatterBase.format_response_payloadN)r2r3r4r5r?rr7__annotations__r@rA staticmethodrLpropertyrr:rPr;rrrTrr9r rWrrrrr>Ls@     r>c@VeZdZdZedeefddZdede dede fdd Z d e dede fd d Z d S)GPT2ContentFormatterzContent handler for GPT2rcCrMr1rNrOrrrrPz(GPT2ContentFormatter.supported_api_typesrBrQrRcC2t|}tdd|dgi|d}t|S)N input_stringrDZinputs parametersr>rLjsondumpsr7encoderrBrQrRrequest_payloadrrrrT  z+GPT2ContentFormatter.format_request_payloadrUc CsTz t|dd}Wntttfy$}z t|jj|d|d}~wwt|dS)Nr0rRtext rcloadsKeyError IndexError TypeErrorrrAformatr rrUrRchoiceerrrrWs z,GPT2ContentFormatter.format_response_payloadNr2r3r4r5rZrr:rPr7rr9rTr rWrrrrr\(  r\c0eZdZUdZdZeed<dfdd ZZS)OSSContentFormatterz`Deprecated: Kept for backwards compatibility Content handler for LLMs from the OSS catalog.Ncontent_formatterrcttddS)Nz`OSSContentFormatter` will be deprecated in the future. Please use `GPT2ContentFormatter` instead. superr warningswarnrO __class__rrr  zOSSContentFormatter.__init__rN r2r3r4r5rzrrXr __classcell__rrrrry  ryc@r[)HFContentFormatterz6Content handler for LLMs from the HuggingFace catalog.rcCrMr1rNrOrrrrPr]z&HFContentFormatter.supported_api_typesrBrQrRcCs.t|td|dg|d}t|S)NrDr`rbrfrrrrTs   z)HFContentFormatter.format_request_payloadrUc CsXz t|ddd}Wntttfy&}z t|jj|d|d}~wwt|dS)Nrrigenerated_textrjrkrmrsrrrrWs z*HFContentFormatter.format_response_payloadNrvrrrrrrwrc@r[)DollyContentFormatterz*Content handler for the Dolly-v2-12b modelrcCrMr1rNrOrrrrPr]z)DollyContentFormatter.supported_api_typesrBrQrRcCr^)Nr_rD) input_datararbrfrrrrTrhz,DollyContentFormatter.format_request_payloadrUc CsPz t|d}Wntttfy"}z t|jj|d|d}~wwt|dS)NrrjrkrmrsrrrrW s z-DollyContentFormatter.format_response_payloadNrvrrrrrrwrc@r[)CustomOpenAIContentFormatterzAContent formatter for models that use the OpenAI like API scheme.rcCs tjtjgSr1)r:r;r=rOrrrrPs z0CustomOpenAIContentFormatter.supported_api_typesrBrQrRcCsrt|}|tjtjfvrtdd|dg|di}n|tjkr,td|i|}ntd|dt |S)z/Formats the request according to the chosen apirrD)r_rarB `api_type` # is not supported by this formatter) r>rLr:r;r<rcrdr=rr7rerfrrrrTs$     z3CustomOpenAIContentFormatter.format_request_payloadrUc Cs|tjtjfvr2z t|dd}Wntttfy,}z t|j j |d|d}~wwt |dS|tj kryzt|dd}t |tsJtdWntttfyd}z t|j j |d|d}~wwt |dt|d |d d d Std |d)zFormats responserrirjNrkchoicesziEndpoint response is not well formed for a chat model. Expected `dict` but `{type(choice)}` was received.rl finish_reasonlogprobs)rr)rlZgeneration_inforr)r:r;r<rcrnrorprqrrArrr r= isinstancedictstripr*rsrrrrW3s>    z4CustomOpenAIContentFormatter.format_response_payloadNrvrrrrrs( rcrx)LlamaContentFormatterzNDeprecated: Kept for backwards compatibility Content formatter for Llama.Nrzrcr{)Nz`LlamaContentFormatter` will be deprecated in the future. Please use `CustomOpenAIContentFormatter` instead. r|rOrrrr \rzLlamaContentFormatter.__init__rrrrrrrUrrc@s8eZdZUdZdZeed< ejZ eed< e dZ e ed< dZ eed< eZeed< dZeed <d Zeed <dZeed < dZeeed < eddZeddededefddZed dededefddZ eddedefddZ!eddededefddZ"ed dddedede#fdd Z$dS)!AzureMLBaseEndpointz Azure ML Online Endpoint models.rrendpoint_api_typerrrN http_client max_retriesrzrQr)Zprotected_namespacesbefore)modevaluesrcCsdtt|dd|d<t|dd|d<t|ddd|d<t|dd tj|d<t|d d tt|d <|S) NrZAZUREML_ENDPOINT_API_KEYrZAZUREML_ENDPOINT_URLrZAZUREML_DEPLOYMENT_NAMErrZAZUREML_ENDPOINT_API_TYPErZAZUREML_TIMEOUT)r rr:r;r7r6)clsrrrrvalidate_environs, z$AzureMLBaseEndpoint.validate_environ field_valuecCs:|d}||jvrtdt|d|jd|d|S)z>Validate that content formatter is supported by endpoint type.rzContent formatter fz8 is not supported by this endpoint. Supported types are z but endpoint is .)r*rPrtype)rrrrrrrvalidate_content_formatters   z.AzureMLBaseEndpoint.validate_content_formattercCs,|dr |dd}|drtd|S)z'Validate that endpoint url is complete./Nzinference.ml.azure.comz`endpoint_url` should contain the full invocation URL including `/score` for `endpoint_api_type='dedicated'` or `/completions` or `/models/chat/completions` for `endpoint_api_type='serverless'`)endswithr)rrrrrvalidate_endpoint_urls   z)AzureMLBaseEndpoint.validate_endpoint_urlcCsZ|d}|tjks|tjkr|dstd|tjkr+|ds+|ds+td|S)zBValidate that endpoint api type is compatible with the URL format.rz/scorezEndpoints of type `dedicated` should follow the format `https://..inference.ml.azure.com/score`. If your endpoint URL ends with `/completions` or`/models/chat/completions`,use `endpoint_api_type='serverless'` instead.z /completionsz/models/chat/completionszEndpoints of type `serverless` should follow the format `https://..inference.ml.azure.com/completions` or `https://..inference.ml.azure.com/models/chat/completions`)r*r:r;r<rrr=)rrrrrrrvalidate_endpoint_api_types"    z.AzureMLBaseEndpoint.validate_endpoint_api_typeT)alwayscCs@|d}|d}|d}|dt}t||||}|S)z?Validate that api key and python package exists in environment.rrrr)r*r6rZget_secret_value)rrrrZ endpoint_keyrrrrrrvalidate_clients    z#AzureMLBaseEndpoint.validate_client)%r2r3r4r5rr7rXr:r;rr rrrr6rr8rrrrzrQrrrZ model_configr classmethodrrrr>rrrrrrrrrresT            rc @speZdZdZedeeeffddZedefddZ  dde ed e e ed e e d ede f d d ZdS)AzureMLOnlineEndpointaAzure ML Online Endpoint models. Example: .. code-block:: python azure_llm = AzureMLOnlineEndpoint( endpoint_url="https://..inference.ml.azure.com/score", endpoint_api_type=AzureMLApiType.dedicated, endpoint_api_key="my-api-key", timeout=120, content_formatter=content_formatter, ) rcCs |jpi}id|jid|iS)zGet the identifying parameters.rrQ)rQr)r _model_kwargsrrr_identifying_paramss z)AzureMLOnlineEndpoint._identifying_paramscCrV)zReturn type of llm.Zazureml_endpointrrOrrr _llm_type szAzureMLOnlineEndpoint._llm_typeNpromptsstopr"r#c Ksv|jpi}|||r||d<g}|D]!}|j|||j}|jj||d} |j| |j} || gqt |dS)anRun the LLM on the given prompts. Args: prompts: The prompt to pass into the model. stop: Optional list of stop words to use when generating. Returns: The string generated by the model. Example: .. code-block:: python response = azureml_model.invoke("Tell me a joke.") r)r!r") generations) rQupdaterzrTrrr0rWappendr ) rrrr"r#rrrBrgZresponse_payloadrrrr _generates"   zAzureMLOnlineEndpoint._generate)NN)r2r3r4r5rZrr7rrrrrr r rrrrrrs&  r),rcurllib.requestr&r~abcrenumrtypingrrrrrZ langchain_core.callbacks.managerr Z#langchain_core.language_models.llmsr Zlangchain_core.outputsr r Zlangchain_core.utilsr rZpydanticrrrrrr6objectrr7r:r>r\ryrrrrrrrrrrs0    / `B