o Zh9@stddlZddlZddlmZmZmZmZmZddlm Z m Z ddl m Z ddl mZddlmZGddde ZdS) N)AnyDictListOptionalcast) AgentAction AgentFinish)BaseCallbackHandler) LLMResult)parsec seZdZUdZdZeed<edZeed<dZeed<dZ eed < d;d ed e ed e ede edd f fdd Z de ee fdeede dd fddZdede dd fddZdede dd fddZdede dd fddZde ee fd e ee fde dd fd!d"Zd#e ee fde dd fd$d%Zdede dd fd&d'Zde ee fd(ede dd fd)d*Zd+ede de fd,d-Z d>> from langchain_community.llms import OpenAI >>> from langchain_community.callbacks import ArgillaCallbackHandler >>> argilla_callback = ArgillaCallbackHandler( ... dataset_name="my-dataset", ... workspace_name="my-workspace", ... api_url="http://localhost:6900", ... api_key="argilla.apikey", ... ) >>> llm = OpenAI( ... temperature=0, ... callbacks=[argilla_callback], ... verbose=True, ... openai_api_key="API_KEY_HERE", ... ) >>> llm.generate([ ... "What is the best NLP-annotation tool out there? (no bias at all)", ... ]) "Argilla, no doubt about it." z%https://github.com/argilla-io/argillaREPO_URLz/issues ISSUES_URLzphttps://docs.argilla.io/en/latest/tutorials_and_integrations/integrations/use_argilla_callback_in_langchain.htmlBLOG_URLzhttp://localhost:6900DEFAULT_API_URLN dataset_nameworkspace_nameapi_urlapi_keyreturnc sLtz ddl}|j|_Wn tytdwt|jtdkr-td|jd|durEtddurEt d|j d |j }|durktd durkt|jtd kr[d nd |_ t d|j d|j }z |j ||dWnty}z td|d|jd|d}~ww||_|p||_z+i}t|jtdkrt d|jdtddi}|jjd*|j|jd||_Wn%ty}ztd|d|jd|jd|jd|jd |d}~wwdd g}|d!d"|jjDkrtd#|jd|jd$|d%d&d"|jjDd'|jd( i|_t d)|jddS)+a)Initializes the `ArgillaCallbackHandler`. Args: dataset_name: name of the `FeedbackDataset` in Argilla. Note that it must exist in advance. If you need help on how to create a `FeedbackDataset` in Argilla, please visit https://docs.argilla.io/en/latest/tutorials_and_integrations/integrations/use_argilla_callback_in_langchain.html. workspace_name: name of the workspace in Argilla where the specified `FeedbackDataset` lives in. Defaults to `None`, which means that the default workspace will be used. api_url: URL of the Argilla Server that we want to use, and where the `FeedbackDataset` lives in. Defaults to `None`, which means that either `ARGILLA_API_URL` environment variable or the default will be used. api_key: API Key to connect to the Argilla Server. Defaults to `None`, which means that either `ARGILLA_API_KEY` environment variable or the default will be used. Raises: ImportError: if the `argilla` package is not installed. ConnectionError: if the connection to Argilla fails. FileNotFoundError: if the `FeedbackDataset` retrieval from Argilla fails. rNzTo use the Argilla callback manager you need to have the `argilla` Python package installed. Please install it with `pip install argilla`z1.8.0z#The installed `argilla` version is z} but `ArgillaCallbackHandler` requires at least version 1.8.0. Please upgrade `argilla` with `pip install --upgrade argilla`.ZARGILLA_API_URLz[Since `api_url` is None, and the env var `ARGILLA_API_URL` is not set, it will default to `z6`, which is the default API URL in Argilla Quickstart.ZARGILLA_API_KEYz1.11.0z admin.apikeyz owner.apikeyz[Since `api_key` is None, and the env var `ARGILLA_API_KEY` is not set, it will default to `z6`, which is the default API key in Argilla Quickstart.)rrz.Could not connect to Argilla with exception: 'z'. Please check your `api_key` and `api_url`, and make sure that the Argilla server is up and running. If the problem persists please report it to z as an `integration` issue.1.14.0zYou have Argilla z., but Argilla 1.14.0 or higher is recommended.Z with_recordsF)nameZ workspacez@`FeedbackDataset` retrieval from Argilla failed with exception `z+`. Please check that the dataset with name=z in the workspace=zz exists in advance. If you need help on how to create a `langchain`-compatible `FeedbackDataset` in Argilla, please visit z.. If the problem persists please report it to promptresponsecSg|]}|jqSr.0fieldrre/var/www/html/lang_env/lib/python3.10/site-packages/langchain_community/callbacks/argilla_callback.py z3ArgillaCallbackHandler.__init__..z`FeedbackDataset` with name=z] had fields that are not supported yet for the`langchain` integration. Supported fields are: z/, and the current `FeedbackDataset` fields are cSrrrrrrr r!r"zl. For more information on how to create a `langchain`-compatible `FeedbackDataset` in Argilla, please visit .zThe `ArgillaCallbackHandler` is currently in beta and is subject to change based on updates to `langchain`. Please report any issues to r)super__init__Zargilla __version__ARGILLA_VERSION ImportErrorr osgetenvwarningswarnrZDEFAULT_API_KEYinit ExceptionConnectionErrorrrZ get_workspacer UserWarningZFeedbackDatasetZ from_argilladatasetFileNotFoundErrorrfields ValueErrorprompts) selfrrrrZrge extra_argsZsupported_fields __class__rr r%=s          zArgillaCallbackHandler.__init__ serializedr5kwargscKs$|jt|dp |d|idS)z.Save the prompts in memory when an LLM starts. parent_run_idrun_idN)r5updatestr)r6r;r5r<rrr on_llm_starts$z#ArgillaCallbackHandler.on_llm_starttokencKdS)z)Do nothing when a new token is generated.Nr)r6rBr<rrr on_llm_new_tokenz'ArgillaCallbackHandler.on_llm_new_tokenrc s|drdS|jt|d}t||jD]\}|jjfdd|Ddq|jt|dt|jtdkrB|j dSdS)z(Log records to Argilla when an LLM ends.r=Nr>cs g|] }d|jdiqS)r3rr)textstrip)rZ generationrrr r!sz5ArgillaCallbackHandler.on_llm_end..recordsr) r5r@zip generationsr1 add_recordspopr r'push_to_argilla)r6rr<r5rMrrIr on_llm_ends  z!ArgillaCallbackHandler.on_llm_enderrorcKrC)z%Do nothing when LLM outputs an error.Nrr6rRr<rrr on_llm_errorrEz#ArgillaCallbackHandler.on_llm_errorinputscKsLd|vr$|jt|dp|dt|dtr|dn|dgidSdS)aIf the key `input` is in `inputs`, then save it in `self.prompts` using either the `parent_run_id` or the `run_id` as the key. This is done so that we don't log the same input prompt twice, once when the LLM starts and once when the chain starts. inputr=r>N)r5r?r@ isinstancelist)r6r;rUr<rrr on_chain_starts  z%ArgillaCallbackHandler.on_chain_startoutputsc s.tfddt|dt|dfDsdSjt|dp.ttjt|dg}|D]+\}}t|trLj j ddt ||Ddq3j j d d || d igdq3t|djvrrjt|dt|djvrjt|dtjtd krj dSdS) zIf either the `parent_run_id` or the `run_id` is in `self.prompts`, then log the outputs to Argilla, and pop the run from `self.prompts`. The behavior differs if the output is a list or not. c3s|]}|jvVqdS)N)r5)rkeyr6rr s  z6ArgillaCallbackHandler.on_chain_end..r=r>NcSs&g|]\}}d||ddiqS)r3rGrF)rH)rroutputrrr r!s z7ArgillaCallbackHandler.on_chain_end..rJr3 rFr)anyr@r5getrritemsrWrXr1rNrLjoinrHrOr r'rP)r6rZr<r5Zchain_output_keyZchain_output_valrr\r on_chain_ends:    z#ArgillaCallbackHandler.on_chain_endcKrC)z+Do nothing when LLM chain outputs an error.NrrSrrr on_chain_error8rEz%ArgillaCallbackHandler.on_chain_error input_strcKrC)zDo nothing when tool starts.Nr)r6r;rfr<rrr on_tool_start<sz$ArgillaCallbackHandler.on_tool_startactioncKrC)z.Do nothing when agent takes a specific action.Nr)r6rhr<rrr on_agent_actionErEz&ArgillaCallbackHandler.on_agent_actionr^observation_prefix llm_prefixcKrC)zDo nothing when tool ends.Nr)r6r^rjrkr<rrr on_tool_endIsz"ArgillaCallbackHandler.on_tool_endcKrC)z&Do nothing when tool outputs an error.NrrSrrr on_tool_errorSrEz$ArgillaCallbackHandler.on_tool_errorrGcKrCz Do nothingNr)r6rGr<rrr on_textWrEzArgillaCallbackHandler.on_textfinishcKrCrnr)r6rpr<rrr on_agent_finish[rEz&ArgillaCallbackHandler.on_agent_finish)NNN)NN)!__name__ __module__ __qualname____doc__r r@__annotations__rrrrr%rrrrArDr rQ BaseExceptionrTrYrdrergrrirlrmrorrq __classcell__rrr9r r s  +       2     r )r)r+typingrrrrrZlangchain_core.agentsrrZlangchain_core.callbacksr Zlangchain_core.outputsr Zpackaging.versionr r rrrr s