o ZhG@sdZddlmZddlZddlmZmZddlmZddl m Z m Z m Z m Z mZddlmZddlmZdd lmZdd lmZdd lmZeeZGd d d eeZGdddeZGdddZGdddeeZ GdddeeZ!GdddeeZ"dS)z3Interfaces to be implemented by general evaluators.) annotationsN)ABCabstractmethod)Enum)AnyOptionalSequenceTupleUnion)warn) AgentAction)BaseLanguageModel)run_in_executor)Chainc@seZdZdZdZ dZ dZ dZ dZ dZ dZ d Z d Z d Z d Z d Z dZ dZ dZ dZ dZ dZ dZ dZdS) EvaluatorTypezThe types of the evaluators.ZqaZcot_qaZ context_qaZpairwise_stringZ score_stringZlabeled_pairwise_stringZlabeled_score_stringZ trajectorycriteriaZlabeled_criteriaZstring_distanceZ exact_matchZ regex_matchZpairwise_string_distanceZembedding_distanceZpairwise_embedding_distanceZ json_validityZ json_equalityZjson_edit_distanceZjson_schema_validationN)__name__ __module__ __qualname____doc__ZQAZCOT_QAZ CONTEXT_QAZPAIRWISE_STRINGZ SCORE_STRINGZLABELED_PAIRWISE_STRINGZLABELED_SCORE_STRINGZAGENT_TRAJECTORYZCRITERIAZLABELED_CRITERIAZSTRING_DISTANCEZ EXACT_MATCHZ REGEX_MATCHZPAIRWISE_STRING_DISTANCEZEMBEDDING_DISTANCEZPAIRWISE_EMBEDDING_DISTANCEZ JSON_VALIDITYZ JSON_EQUALITYZJSON_EDIT_DISTANCEZJSON_SCHEMA_VALIDATIONrrR/var/www/html/lang_env/lib/python3.10/site-packages/langchain/evaluation/schema.pyrsTrc@s"eZdZdZeed ddZd S) LLMEvalChainz,A base class for evaluators that use an LLM.llmr kwargsrreturncKdS)z#Create a new evaluator from an LLM.Nr)clsrrrrrfrom_llmMzLLMEvalChain.from_llmN)rr rrrr)rrrr classmethodrrrrrrrJs rc@sXeZdZdZedddZedddZedd d Zedd d Z ddddZ d S)_EvalArgsMixinz(Mixin for checking evaluation arguments.rboolcCrz2Whether this evaluator requires a reference label.Frselfrrrrequires_referenceVz!_EvalArgsMixin.requires_referencecCr)0Whether this evaluator requires an input string.Frr$rrrrequires_input[r'z_EvalArgsMixin.requires_inputstrcCd|jjdS)z&Warning to show when input is ignored.zIgnoring input in , as it is not expected. __class__rr$rrr_skip_input_warning`sz"_EvalArgsMixin._skip_input_warningcCr+)z*Warning to show when reference is ignored.zIgnoring reference in r,r-r$rrr_skip_reference_warningesz&_EvalArgsMixin._skip_reference_warningN reference Optional[str]inputNonecCs||jr|durt|jjd|dur|jst|j|jr,|dur,t|jjd|dur:|jsr1r?r3r?rrrr@) r=r*r1r2r3r2rrrr@) rrrrr9r;r&rrBrDrErFrrrrr:s&  #r:c@s\eZdZdZeddddddZddddddZddddddZddddddZdS)PairwiseStringEvaluatorzDCompare the output of two models (or two outputs of the same model).Nr<r=r* prediction_br1r2r3rrrr@cKr)1Evaluate the output string pairs. Args: prediction (str): The output string from the first model. prediction_b (str): The output string from the second model. reference (Optional[str], optional): The expected output / reference string. input (Optional[str], optional): The input string. kwargs: Additional keyword arguments, such as callbacks and optional reference strings. Returns: dict: A dictionary containing the preference, scores, and/or other information. Nrr%r=rHr1r3rrrr_evaluate_string_pairsrz.PairwiseStringEvaluator._evaluate_string_pairscs&td|jf||||d|IdHS)@Asynchronously evaluate the output string pairs. Args: prediction (str): The output string from the first model. prediction_b (str): The output string from the second model. reference (Optional[str], optional): The expected output / reference string. input (Optional[str], optional): The input string. kwargs: Additional keyword arguments, such as callbacks and optional reference strings. Returns: dict: A dictionary containing the preference, scores, and/or other information. Nr=rHr1r3)rrKrJrrr_aevaluate_string_pairss z/PairwiseStringEvaluator._aevaluate_string_pairscKs(|j||d|jd||||d|S)rIr<rMNr)r6rKrJrrrevaluate_string_pairs/sz-PairwiseStringEvaluator.evaluate_string_pairscs0|j||d|jd||||d|IdHS)rLr<rMNr)r6rNrJrrraevaluate_string_pairsLs z.PairwiseStringEvaluator.aevaluate_string_pairs) r=r*rHr*r1r2r3r2rrrr@) rrrrrrKrNrOrPrrrrrGs#"rGc@sbeZdZdZedddZedddddZdddddZdddddZ dddddZ dS)AgentTrajectoryEvaluatorz,Interface for evaluating agent trajectories.rr"cCr)r(Trr$rrrr)mr'z'AgentTrajectoryEvaluator.requires_inputN)r1r=r*agent_trajectory!Sequence[Tuple[AgentAction, str]]r3r1r2rrr@cKr)Evaluate a trajectory. Args: prediction (str): The final predicted response. agent_trajectory (List[Tuple[AgentAction, str]]): The intermediate steps forming the agent trajectory. input (str): The input to the agent. reference (Optional[str]): The reference answer. Returns: dict: The evaluation result. Nrr%r=rRr3r1rrrr_evaluate_agent_trajectoryrrz3AgentTrajectoryEvaluator._evaluate_agent_trajectorycs&td|jf||||d|IdHS)Asynchronously evaluate a trajectory. Args: prediction (str): The final predicted response. agent_trajectory (List[Tuple[AgentAction, str]]): The intermediate steps forming the agent trajectory. input (str): The input to the agent. reference (Optional[str]): The reference answer. Returns: dict: The evaluation result. N)r=rRr1r3)rrVrUrrr_aevaluate_agent_trajectorys z4AgentTrajectoryEvaluator._aevaluate_agent_trajectorycKs(|j||d|jd||||d|S)rTr<r=r3rRr1Nr)r6rVrUrrrevaluate_agent_trajectorysz2AgentTrajectoryEvaluator.evaluate_agent_trajectorycs0|j||d|jd||||d|IdHS)rWr<rYNr)r6rXrUrrraevaluate_agent_trajectorys z3AgentTrajectoryEvaluator.aevaluate_agent_trajectoryr7) r=r*rRrSr3r*r1r2rrrr@) rrrrr9r)rrVrXrZr[rrrrrQjs %$rQ)#r __future__rloggingabcrrenumrtypingrrrr r warningsr Zlangchain_core.agentsr Zlangchain_core.language_modelsr Zlangchain_core.runnables.configrZlangchain.chains.baser getLoggerrloggerr*rrr!r:rGrQrrrrs$        6 1tr