o Zh@s>dZddlmZddlZddlZddlZddlmZddlm Z m Z ddl m Z m Z mZmZmZmZddlmZddlmZdd lmZzdd lmZmZmZmZWneygdd lmZmZmZmZYnwddlZdd l m!Z!dd l"m#Z#m$Z$m%Z%m&Z&e'e(Z)Gd ddeZ*GdddeddZ+GdddeZ,GdddeddZ-GdddZ.ee,e-e/fZ0GdddeZ1ee1e/fZ2Gddde.Z3d;dd Z4d!Z5dd.d/Z9d?d1d2Z:d@d6d7Z;ee e ej&e ej%gee,e-ffe edS)Bz?This module contains the evaluator classes for evaluating runs.) annotationsN)abstractmethod) AwaitableSequence)AnyCallableLiteralOptionalUnioncast) TypedDict run_helpers)schemas) BaseModelFieldValidationError validator)wraps) SCORE_TYPE VALUE_TYPEExampleRunc@s$eZdZUdZded< ded<dS)Categoryz$A category for categorical feedback.Optional[Union[float, int]]valuestrlabelN__name__ __module__ __qualname____doc____annotations__r$r$U/var/www/html/lang_env/lib/python3.10/site-packages/langsmith/evaluation/evaluator.pyr/s rc@s8eZdZUdZded< ded< ded< ded<d S) FeedbackConfigziConfiguration to define a type of feedback. Applied on on the first creation of a feedback_key. z0Literal['continuous', 'categorical', 'freeform']typerminmaxz%Optional[list[Union[Category, dict]]] categoriesNrr$r$r$r%r&8s  r&F)totalc@seZdZUdZded< dZded< dZded< dZd ed < dZd ed < e e d Z ded< dZ ded< dZ ded< dZded< dZd ed< GdddZedddddZdS)EvaluationResultzEvaluation result.rkeyNrscorerrz Optional[str]commentzOptional[dict] correction)default_factorydictevaluator_infoz%Optional[Union[FeedbackConfig, dict]]feedback_configOptional[Union[uuid.UUID, str]] source_run_id target_run_idextrac@seZdZdZdZdS)zEvaluationResult.ConfigzPydantic model configuration.FN)rr r!r"Z allow_extrar$r$r$r%Configbsr9T)precCs6d|vs |ddurt|ttfrtd||S)z$Check that the value is not numeric.r.NzJNumeric values should be provided in the 'score' field, not 'value'. Got: ) isinstanceintfloatloggerwarning)clsvvaluesr$r$r%check_value_non_numericgsz(EvaluationResult.check_value_non_numeric)rr r!r"r#r.rr/r0rr2r3r4r6r7r8r9rrCr$r$r$r%r,Gs2          r,c@seZdZUdZded<dS)EvaluationResultszqBatch evaluation results. This makes it easy for your evaluator to return multiple metrics at once. zlist[EvaluationResult]resultsNrr$r$r$r%rDvs rDc@s4eZdZdZe  ddd d Z  ddd dZdS) RunEvaluatorzEvaluator interface class.NrunrexampleOptional[Example]evaluator_run_idOptional[uuid.UUID]return*Union[EvaluationResult, EvaluationResults]cCsdS)zEvaluate an example.Nr$)selfrGrHrJr$r$r% evaluate_runszRunEvaluator.evaluate_runcs4tfdd}td|IdHS)z#Evaluate an example asynchronously.csBtjdiWdS1swYdS)Nr$)rhZtracing_contextrOr$Zcurrent_contextrJrHrGrNr$r%_run_with_contexts $z5RunEvaluator.aevaluate_run.._run_with_contextN)rPZget_tracing_contextasyncioZget_running_loopZrun_in_executor)rNrGrHrJrRr$rQr% aevaluate_runszRunEvaluator.aevaluate_runNNrGrrHrIrJrKrLrM)rr r!r"rrOrTr$r$r$r%rFs rFc@s@eZdZUdZded< ded< dZded< dZd ed <dS) ComparisonEvaluationResultzFeedback scores for the results of comparative evaluations. These are generated by functions that compare two or more runs, returning a ranking or other feedback. rr-z'dict[Union[uuid.UUID, str], SCORE_TYPE]scoresNr5r6z6Optional[Union[str, dict[Union[uuid.UUID, str], str]]]r/)rr r!r"r#r6r/r$r$r$r%rWs   rWcseZdZdZ d-d.ddZ d/d0ddZd1ddZd2ddZed3ddZ  d4d5d$d%Z  d4d6fd&d' Z d-d7d(d)Z d8d+d,Z ZS)9DynamicRunEvaluatoraA dynamic evaluator that wraps a function and transforms it into a `RunEvaluator`. This class is designed to be used with the `@run_evaluator` decorator, allowing functions that take a `Run` and an optional `Example` as arguments, and return an `EvaluationResult` or `EvaluationResults`, to be used as instances of `RunEvaluator`. Attributes: func (Callable): The function that is wrapped by this evaluator. NfuncXCallable[[Run, Optional[Example]], Union[_RUNNABLE_OUTPUT, Awaitable[_RUNNABLE_OUTPUT]]]afuncIOptional[Callable[[Run, Optional[Example]], Awaitable[_RUNNABLE_OUTPUT]]]cst|\}|rt|\}d fdd }t||ddlm}|dur4|j||d |_t|d d |_t |rR|durAt d |j||d |_t|d d |_dS|jt t t ttgtf||d |_t|d d |_dS)zInitialize the DynamicRunEvaluator with a given function. Args: func (Callable): A function that takes a `Run` and an optional `Example` as arguments, and returns a dict or `ComparisonEvaluationResult`. inputsr2rLc,dur|S|d|d\}}}|S)NrGrHgetr^_ traced_inputsZprepare_inputsr$r%process_inputs  z4DynamicRunEvaluator.__init__..process_inputsrr NrfrrYFunc was provided as a coroutine function, but afunc was also provided. If providing both, func should be a regular function to avoid ambiguity.r^r2rLr2)_normalize_evaluator_funcr langsmithrensure_traceabler\getattr_nameinspectiscoroutinefunction TypeErrorr rrr r_RUNNABLE_OUTPUTrZrNrZr\rfrr$rer%__init__s2     zDynamicRunEvaluator.__init__FresultUnion[EvaluationResult, dict]r6 uuid.UUID allow_no_keyboolrLr,c sttr js |_Sz2stddvr"|r"|jd<tfdddDr4tdtd id|iWStyS}ztd|d}~ww) NziExpected an EvaluationResult object, or dict with a metric 'key' and optional 'score'; got empty result: r-c3s|]}|vVqdSNr$).0krvr$r% z@DynamicRunEvaluator._coerce_evaluation_result..)r.rr/zrExpected an EvaluationResult object, or dict with a metric 'key' and optional 'score' or categorical 'value'; got r6z[Expected an EvaluationResult object, or dict with a metric 'key' and optional 'score'; got r$)r;r,r6 ValueErrorroallr)rNrvr6ryer$r~r%_coerce_evaluation_results<   z-DynamicRunEvaluator._coerce_evaluation_resultrEUnion[dict, EvaluationResults]rMcsPd|vr|}fdd|dD|d<tdi|Sjtt|ddS)NrEcsg|] }j|dqS))r6)r)r|rrNr6r$r% s zBDynamicRunEvaluator._coerce_evaluation_results..T)r6ryr$)copyrDrr r2)rNrEr6cpr$rr%_coerce_evaluation_resultss   z.DynamicRunEvaluator._coerce_evaluation_resultsMUnion[EvaluationResult, EvaluationResults, dict, str, int, bool, float, list]cCs.t|tr |js ||_|St|}|||Sr{)r;r,r6_format_evaluator_resultr)rNrvr6r$r$r%_format_result)s  z"DynamicRunEvaluator._format_resultcC t|dSzCheck if the evaluator function is asynchronous. Returns: bool: True if the evaluator function is asynchronous, False otherwise. r\hasattrrNr$r$r%is_async7 zDynamicRunEvaluator.is_asyncrGrrHrIrJrKcCst|dst}|rtd||||S|dur"t}d|j i}t |ddr4t |j |d<|j ||||dd}|||S) aEvaluate a run using the wrapped function. This method directly invokes the wrapped function with the provided arguments. Args: run (Run): The run to be evaluated. example (Optional[Example]): An optional example to be used in the evaluation. Returns: Union[EvaluationResult, EvaluationResults]: The result of the evaluation. rZtCannot call `evaluate_run` on an async run evaluator from within an running event loop. Use `aevaluate_run` instead.Nr7 session_id experimentrun_idmetadataZlangsmith_extra)rrSget_event_loop is_running RuntimeErrorrun_until_completerTuuiduuid4idrnrrrZr)rNrGrHrJ running_looprrvr$r$r%rO@s$    z DynamicRunEvaluator.evaluate_runcs|t|dst||IdHS|durt}d|ji}t|ddr*t|j|d<|j ||||ddIdH}| ||S)aEvaluate a run asynchronously using the wrapped async function. This method directly invokes the wrapped async function with the provided arguments. Args: run (Run): The run to be evaluated. example (Optional[Example]): An optional example to be used in the evaluation. Returns: Union[EvaluationResult, EvaluationResults]: The result of the evaluation. r\Nr7rrrr) rsuperrTrrrrnrrr\r)rNrGrHrJrrv __class__r$r%rTfs    z!DynamicRunEvaluator.aevaluate_runcC |||S)aMake the evaluator callable, allowing it to be used like a function. This method enables the evaluator instance to be called directly, forwarding the call to `evaluate_run`. Args: run (Run): The run to be evaluated. example (Optional[Example]): An optional example to be used in the evaluation. Returns: Union[EvaluationResult, EvaluationResults]: The result of the evaluation. )rO)rNrGrHr$r$r%__call__ zDynamicRunEvaluator.__call__rcCd|jdS))Represent the DynamicRunEvaluator object.zrorr$r$r%__repr__zDynamicRunEvaluator.__repr__r{)rZr[r\r])F)rvrwr6rxryrzrLr,)rErr6rxrLrM)rvrr6rxrLrMrLrzrUrV)rGrrHrIrJrK)rGrrHrIrLrMrLr)rr r!r"rurrrpropertyrrOrTrr __classcell__r$r$rr%rYs& >     )" rYrZr[cCt|S)zmCreate a run evaluator from a function. Decorator that transforms a function into a `RunEvaluator`. )rYrZr$r$r% run_evaluators ri'objrcCs,t|}t|tkr|dtdd}|S)Nz...))reprlen_MAXSIZE)rsr$r$r% _maxsize_reprs rc@sxeZdZdZ d$d%ddZed&d d Z d$d'ddZ d$d'ddZ d$d'ddZ d(ddZ e d)ddZ d*d"d#Z dS)+DynamicComparisonRunEvaluatorz4Compare predictions (as traces) from 2 or more runs.NrZfCallable[[Sequence[Run], Optional[Example]], Union[_COMPARISON_OUTPUT, Awaitable[_COMPARISON_OUTPUT]]]r\UOptional[Callable[[Sequence[Run], Optional[Example]], Awaitable[_COMPARISON_OUTPUT]]]cst|\}|rt|\}d fdd }t||ddlm}|dur4|j||d |_t|d d |_t |rR|durAt d |j||d |_t|d d |_dS|jt t t tttgtf||d |_t|d d |_dS)zInitialize the DynamicRunEvaluator with a given function. Args: func (Callable): A function that takes a `Run` and an optional `Example` as arguments, and returns an `EvaluationResult` or `EvaluationResults`. r^r2rLcr_)NrunsrHr`rbrer$r%rfrgz>DynamicComparisonRunEvaluator.__init__..process_inputsrr NrhrrYrirj)$_normalize_comparison_evaluator_funcrrlrrmr\rnrorprqrrr rrrr r_COMPARISON_OUTPUTrZrtr$rer%rus@      z&DynamicComparisonRunEvaluator.__init__rLrzcCrrrrr$r$r%rrz&DynamicComparisonRunEvaluator.is_asyncr Sequence[Run]rHrIrWcCsjt|dst}|rtd||||St}| |}|j ||||dd}| |||S)zCompare runs to score preferences. Args: runs: A list of runs to compare. example: An optional example to be used in the evaluation. rZrrtagsr) rrSrrrr acompare_runsrr _get_tagsrZ_format_results)rNrrHrr6rrvr$r$r% compare_runss"   z*DynamicComparisonRunEvaluator.compare_runscsTt|ds |||St}||}|j||||ddIdH}||||S)aEvaluate a run asynchronously using the wrapped async function. This method directly invokes the wrapped async function with the provided arguments. Args: runs (Run): The runs to be evaluated. example (Optional[Example]): An optional example to be used in the evaluation. Returns: ComparisonEvaluationResult: The result of the evaluation. r\rrN)rrrrrr\r)rNrrHr6rrvr$r$r%rs    z+DynamicComparisonRunEvaluator.acompare_runscCr)aMake the evaluator callable, allowing it to be used like a function. This method enables the evaluator instance to be called directly, forwarding the call to `evaluate_run`. Args: run (Run): The run to be evaluated. example (Optional[Example]): An optional example to be used in the evaluation. Returns: ComparisonEvaluationResult: The result of the evaluation. )r)rNrrHr$r$r%r:rz&DynamicComparisonRunEvaluator.__call__rcCr)rzfzADynamicComparisonRunEvaluator._format_results..)rXr-r6r-zXExpected 'dict', 'list' or 'ComparisonEvaluationResult' result object. Received: result=r6zExpected a dictionary with a 'key' and dictionary of scores mappingrun IDs to numeric scores, or ComparisonEvaluationResult object, got r$) r;rWr6listzipror2rr)rNrvr6rmsgrr$r$r%rZs@     z-DynamicComparisonRunEvaluator._format_resultsr{)rZrr\rr)rrrHrIrLrWr)rrrLr)rvrr6rxrrrLrW)rr r!r"rurrrrrr staticmethodrrr$r$r$r%rs  @      rrrLcCr)z.Create a comaprison evaluator from a function.)rrr$r$r%comparison_evaluatorsrrtuple[Union[Callable[[Run, Optional[Example]], _RUNNABLE_OUTPUT], Callable[[Run, Optional[Example]], Awaitable[_RUNNABLE_OUTPUT]]], Optional[Callable[..., dict]]]c:dtddjD}ddjD|r6tfdd|Ds@tfdd|Ddkr@d d }t|tfd d|DrR|d d gkrVdfStr|dfdd dfdd }tdrtt dn|j |_ |fSdfdd dfdd }tdrt dn|j |_ |fS)N)rGrHr^outputsreference_outputs attachmentscS g|] \}}|j|jkr|qSr$kind VAR_KEYWORDr|pnamepr$r$r%r z-_normalize_evaluator_func..cS"g|] \}}|jtjjur|qSr$defaultrp Parameteremptyrr$r$r%r c3 |] }|vp |vVqdSr{r$r|rargs_with_defaultssupported_argsr$r%r z,_normalize_evaluator_func..cg|]}|vr|qSr$r$r|arr$r%rrUInvalid evaluator function. Must have at least one argument. Supported arguments are . Please see https://docs.smith.langchain.com/evaluation/how_to_guides/evaluation/evaluate_llm_application#use-custom-evaluatorsc3rr{r$rrr$r%rrrGrHrrIrLtuple[list, dict, dict]c|||r|jni|jp i|r|jpini|r|jpinid}i}g}i}jD]/\}}||vrY|j|j|jfvrC|||n||||<|dvrSt ||n||||<q*|||fSN)rGrHr^rrr)rGrH r^rr parametersitemsrPOSITIONAL_OR_KEYWORDPOSITIONAL_ONLYrrrGrHarg_mapkwargsargsrd param_nameparamsigr$r%_prepare_inputs8      z2_normalize_evaluator_func.._prepare_inputsrsc&||\}}}|i|IdHSr{r$rGrHrrrcrrZr$r%awrapperz+_normalize_evaluator_func..awrapperrcrrrrrr$r%rrc||\}}}|i|Sr{r$r r r$r%wrappersz*_normalize_evaluator_func..wrapper)rGrrHrIrLr)rGrrHrIrLrs rp signaturerrrrrrqrrnrrZall_argsrr rr$rrrZrrr%rksP    rktuple[Union[Callable[[Sequence[Run], Optional[Example]], _COMPARISON_OUTPUT], Callable[[Sequence[Run], Optional[Example]], Awaitable[_COMPARISON_OUTPUT]]], Optional[Callable[..., dict]]]cr)NrrHr^rrcSrr$rrr$r$r%rrz8_normalize_comparison_evaluator_func..cSrr$rrr$r$r%rrc3rr{r$rrr$r%r%rz7_normalize_comparison_evaluator_func..crr$r$rrr$r%r(rrrrc3rr{r$rrr$r%r3rrrHrrIrLrc|||r|jnidd|D|r|jpinid}i}g}i}jD]/\}}||vrS|j|j|jfvr=|||n||||<|dvrMt||n||||<q$|||fS)NcSg|]}|jpiqSr$rr|rGr$r$r%rDQ_normalize_comparison_evaluator_func.._prepare_inputs..rrrH r^rrrrrrrrrrHrrrrdrrrr$r%r=2      z=_normalize_comparison_evaluator_func.._prepare_inputsrcrr{r$rrHrrrcr r$r%r [r z6_normalize_comparison_evaluator_func..awrapperrcr)NcSrr$rrr$r$r%rqrrrrrrrr$r%rjrcr r{r$r r r$r%rsz5_normalize_comparison_evaluator_func..wrapper)rrrHrIrLr)rrrHrIrLrrrr$rr%rsP    rrv;Union[EvaluationResults, dict, str, int, bool, float, list]Union[EvaluationResults, dict]cCst|tttfrd|i}|S|std|t|tr3tdd|Ds-td|dd|i}|St|tr>d|i}|St|trF |Std |) Nr.zdExpected a non-empty dict, str, bool, int, float, list, EvaluationResult, or EvaluationResults. Got css|]}t|tVqdSr{)r;r2)r|xr$r$r%rsz+_format_evaluator_result..z8Expected a list of dicts or EvaluationResults. Received .rErzZExpected a dict, str, bool, int, float, list, EvaluationResult, or EvaluationResults. Got ) r;rzr=r<rrrrr2r~r$r$r%rs6    rSUMMARY_EVALUATOR_TcsdtddjD}ddjD|r6tfdd|DsJtfdd|DdkrJd d }|rF|d |d 7}t|tfd d|Dr[|d dgkr]Sdfdd }tdrrtd|_ |S|j |_ |S)Nrexamplesr^rrcSsg|]\}}|qSr$r$rr$r$r%rsz0_normalize_summary_evaluator..cSrr$rrr$r$r%rrc3rr{r$rrr$r%rrz/_normalize_summary_evaluator..crr$r$rrr$r%rrrrr$z Received arguments c3s|]}|vVqdSr{r$r)rr$r%rrrr'Sequence[schemas.Run]Sequence[schemas.Example]rLrMcs||dd|Ddd|Ddd|Dd}i}g}jD]\}}||vr?|j|j|jfvr9|||q ||||<q |i|}t|trN|St|S)NcSsg|]}|jqSr$)r^r|rHr$r$r%rszA_normalize_summary_evaluator..wrapper..cSrr$rrr$r$r%rrcSrr$rr*r$r$r%rrr&) rrrrrrr;r,r)rr'rrrrrrv)rZrr$r%rs*     z-_normalize_summary_evaluator..wrapperr)rr(r'r)rLrM) rprrrrrrrrnr)rZrrrr$)rrZrrr%_normalize_summary_evaluators> r+)rZr[)rr)rZrrLr)rZrrLr)rZrrLr)rvr!rLr")rZrrLr%)?r" __future__rrSrprabcrcollections.abcrrtypingrrrr r r typing_extensionsr rlrrPrZ pydantic.v1rrrr ImportErrorZpydanticlogging functoolsrZlangsmith.schemasrrrr getLoggerrr>rr&r,rDrFr2rsrWrrYrrrrrrkrrrr%r+r$r$r$r%sn         /   i  M