o Zh3@s*dZddlZddlZddlmZddlmZmZmZm Z m Z m Z m Z m Z mZddlmZddlmZddlmZddlmZdd lmZdd lmZe d eee efd Ze d eee e ee fd ZGdddZGdddeeefZ Gddde eefZ!eddddGdddeZ"dS)zSagemaker InvokeEndpoint API.N)abstractmethod) AnyDictGenericIteratorListMappingOptionalTypeVarUnion) deprecated)CallbackManagerForLLMRun)LLM)pre_init) ConfigDict)enforce_stop_tokens INPUT_TYPE)bound OUTPUT_TYPEc@s:eZdZdZdeddfddZd ddZdefd d ZdS) LineIteratoraParse the byte stream input. The output of the model will be in the following format: b'{"outputs": [" a"]} ' b'{"outputs": [" challenging"]} ' b'{"outputs": [" problem"]} ' ... While usually each PayloadPart event from the event stream will contain a byte array with a full json, this is not guaranteed and some of the json objects may be split acrossPayloadPart events. For example: {'PayloadPart': {'Bytes': b'{"outputs": '}} {'PayloadPart': {'Bytes': b'[" problem"]} '}} This class accounts for this by concatenating bytes written via the 'write' function and then exposing a method which will return lines (ending with a ' ' character) within the buffer via the 'scan_lines' function. It maintains the position of the last read position to ensure that previous bytes are not exposed again. For more details see: https://aws.amazon.com/blogs/machine-learning/elevating-the-generative-ai-experience-introducing-streaming-support-in-amazon-sagemaker-hosting/ streamreturnNcCst||_t|_d|_dS)Nr)iter byte_iteratorioBytesIObufferread_pos)selfrrb/var/www/html/lang_env/lib/python3.10/site-packages/langchain_community/llms/sagemaker_endpoint.py__init__2s   zLineIterator.__init__cCs|S)Nrrrrr __iter__7szLineIterator.__iter__cCs |j|j|j}|r&|dtdkr&|jt|7_|ddSzt|j}Wnty@|j|j j kr?Yqwd|vrFq|jdt j |j |ddq)NT Z PayloadPartrBytes)rseekrreadlineordlennextr StopIteration getbuffernbytesrSEEK_ENDwrite)rlinechunkrrr __next__:s$   zLineIterator.__next__)rr)__name__ __module__ __qualname____doc__rr!r#r3rrrr rs  rc@sheZdZUdZ dZeeed< dZeeed< e de de de fdd Z e d e defd d Zd S)ContentHandlerBasezHandler class to transform input from LLM to a format that SageMaker endpoint expects. Similarly, the class handles transforming output from the SageMaker endpoint to a format that LLM class expects. z text/plain content_typeacceptsprompt model_kwargsrcCdS)zTransforms the input to a format that model can accept as the request Body. Should return bytes or seekable file like object in the format specified in the content_type request header. Nr)rr;r<rrr transform_inputmz"ContentHandlerBase.transform_inputoutputcCr=)z[Transforms the output from the model to string that the LLM class expects. Nr)rr@rrr transform_outputur?z#ContentHandlerBase.transform_outputN)r4r5r6r7r9r str__annotations__r:rrrbytesr>rrArrrr r8Ns r8c@seZdZdZdS)LLMContentHandlerzContent handler for LLM class.N)r4r5r6r7rrrr rE|srEz0.3.16z1.0z$langchain_aws.llms.SagemakerEndpoint)ZsinceZremovalZalternative_importc @s eZdZUdZ dZeed< dZeed< dZ eed< dZ e eed< e ed< d Z eed < dZe eed < dZe eed < ed dZededefddZedeeeffddZedefddZ  ddede eede ededef ddZdS)SagemakerEndpointaSagemaker Inference Endpoint models. To use, you must supply the endpoint name from your deployed Sagemaker model & the region where it is deployed. To authenticate, the AWS client uses the following methods to automatically load credentials: https://boto3.amazonaws.com/v1/documentation/api/latest/guide/credentials.html If a specific credential profile should be used, you must pass the name of the profile from the ~/.aws/credentials file that is to be used. Make sure the credentials / roles used have the required policies to access the Sagemaker endpoint. See: https://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies.html Nclient endpoint_name region_namecredentials_profile_namecontent_handlerF streamingr<endpoint_kwargsZforbid)extravaluesrc Cs|ddur |S z8ddl}z"|ddur|j|dd}n|}|jd|dd|d<WW|StyB}ztd |d}~wwtyMtd w) z.Dont do anything if client provided externallyrGNrrK)Z profile_namezsagemaker-runtimerJ)rJzCould not load credentials to authenticate with AWS client. Please check that credentials in the specified profile name are valid.zRCould not import boto3 python package. Please install it with `pip install boto3`.)getboto3SessionrG Exception ValueError ImportError)clsrPrRsessionerrr validate_environments8  z&SagemakerEndpoint.validate_environmentcCs |jpi}id|jid|iS)zGet the identifying parameters.rIr<)r<rI)r _model_kwargsrrr _identifying_params's z%SagemakerEndpoint._identifying_paramscCr=)zReturn type of llm.Zsagemaker_endpointrr"rrr _llm_type0szSagemakerEndpoint._llm_typer;stop run_managerkwargsc Ks\|jpi}i||}|jpi}|j||}|jj}|jj} |jrv|rvz>|jjd |j ||jjd|} t | d} d} | D] } t | } | dd}|durVt||}| |7} ||q?| WStyu}ztd|d}~wwz|jjd |j ||| d|}Wnty}ztd |d}~ww|j|d}|durt||}|S) afCall out to Sagemaker inference endpoint. Args: prompt: The prompt to pass into the model. stop: Optional list of stop words to use when generating. Returns: The string generated by the model. Example: .. code-block:: python response = se("Tell me a joke.") ) EndpointNameBody ContentTyperbrHZoutputsrNz.Error raised by streaming inference endpoint: )rarbrcAcceptz$Error raised by inference endpoint: r)r<rNrLr>r9r:rMrGZ$invoke_endpoint_with_response_streamrIrjsonloadsrQrZon_llm_new_tokenrTrUZinvoke_endpointrA)rr;r^r_r`r[Z_endpoint_kwargsbodyr9r:respiteratorZcurrent_completionr1Z resp_outputrYresponsetextrrr _call5s\          zSagemakerEndpoint._call)NN)r4r5r6r7rGrrCrIrBrJrKr rErMboolr<rrNrZ model_configrrZpropertyrr\r]rr rlrrrr rFsT  0   $ rF)#r7rreabcrtypingrrrrrrr r r Zlangchain_core._api.deprecationr Zlangchain_core.callbacksr Z#langchain_core.language_models.llmsrZlangchain_core.utilsrZpydanticrZlangchain_community.llms.utilsrrBrfloatrrr8rErFrrrr s, ,      :.