o Zh7*@sdZddlmZddlZddlZddlmZmZmZm Z m Z ddl m Z ddl mZddlmZddlmZdd lmZdd lmZmZmZdd lmZmZdd lmZdddZdddZGdddeeeZ GdddeeeZ!Gddde!Z"dS)z-LLM Chains for evaluating question answering.) annotationsN)AnyListOptionalSequenceTuple) Callbacks)BaseLanguageModel)PromptTemplate) ConfigDict)LLMChain)CONTEXT_PROMPT COT_PROMPTPROMPT) LLMEvalChainStringEvaluator)RUN_KEYtextstrreturnOptional[Tuple[str, int]]cCstd|tj}|r"|ddkrdS|ddkr"dSzI|dt ddt j }|dkr=WdS|dkrFWdS|d t ddt j }|dkr`WdS|dkriWdSWdSt yuYdSw) Nzgrade:\s*(correct|incorrect)CORRECT)rr INCORRECT)rrr) researchstrip IGNORECASEgroupuppersplit translater maketransstring punctuation IndexError)rmatchZ first_word last_wordr*Y/var/www/html/lang_env/lib/python3.10/site-packages/langchain/evaluation/qa/eval_chain.py _get_scores8      r,dictcCs6|}t|}|durd\}}n|\}}|||dS)zParse the output text. Args: text (str): The output text to parse. Returns: Any: The parsed output. N)NN) reasoningvaluescore)rr,)rr.Z parsed_scoresr/r0r*r*r+_parse_string_eval_output1s  r1c@seZdZUdZdZded<eddZed7d d Z e d8d d Z e d7d dZ e d7ddZ e d9d:ddZ   d;dddd3d4Zdddd,d-d>d5d6ZdS)? QAEvalChainz,LLM Chain for evaluating question answering.resultsr output_keyignoreextrarboolcCdSNFr*clsr*r*r+is_lc_serializablePzQAEvalChain.is_lc_serializablecCr9)NZ correctnessr*selfr*r*r+evaluation_nameTr>zQAEvalChain.evaluation_namecCr9NTr*r?r*r*r+requires_referenceXr>zQAEvalChain.requires_referencecCr9rBr*r?r*r*r+requires_input\r>zQAEvalChain.requires_inputNllmr promptOptional[PromptTemplate]kwargsrcKsH|pt}hd}|t|jkrtd|d|j|d||d|S)aLoad QA Eval Chain from LLM. Args: llm (BaseLanguageModel): the base language model to use. prompt (PromptTemplate): A prompt template containing the input_variables: 'input', 'answer' and 'result' that will be used as the prompt for evaluation. Defaults to PROMPT. **kwargs: additional keyword arguments. Returns: QAEvalChain: the loaded QA eval chain. >queryanswerresultInput variables should be , but got rErFNr*)rsetinput_variables ValueError)r<rErFrHexpected_input_varsr*r*r+from_llm`szQAEvalChain.from_llmrIrJrK callbacksexamplesSequence[dict] predictions question_key answer_keyprediction_keyrUr List[dict]c*fddt|D}|j||dS)5Evaluate question answering examples and predictions.c,g|]\}}|||dqS)rIrJrKr*.0iZexamplerZr[rXrYr*r+  z(QAEvalChain.evaluate..rT enumerateapply)r@rVrXrYrZr[rUinputsr*rdr+evaluate  zQAEvalChain.evaluater-cC&t||j}t|vr|t|t<|SNr1r4rr@rK parsed_resultr*r*r+_prepare_output zQAEvalChain._prepare_outputF referenceinputrUinclude_run_info predictionru Optional[str]rvrwcK ||||d||d}||S)aEvaluate Chain or LLM output, based on optional input and label. Args: prediction (str): the LLM or chain prediction to evaluate. reference (Optional[str], optional): the reference label to evaluate against. input (Optional[str], optional): the input to consider during evaluation callbacks (Callbacks, optional): the callbacks to use for tracing. include_run_info (bool, optional): whether to include run info in the returned results. **kwargs: additional keyword arguments, including callbacks, tags, etc. Returns: dict: The evaluation results containing the score or value. r`rUrwrrr@rxrurvrUrwrHrKr*r*r+_evaluate_stringss zQAEvalChain._evaluate_stringsc*|j|||d||dIdH}||S)Nr`rjrUrwZacallrrr}r*r*r+_aevaluate_strings  zQAEvalChain._aevaluate_stringsrr8rrrn)rEr rFrGrHrrr2r`)rVrWrXrWrYrrZrr[rrUrrr\rKr-rr-rxrruryrvryrUrrwr8rHrrr-)__name__ __module__ __qualname____doc__r4__annotations__r model_config classmethodr=propertyrArCrDrSrkrrr~rr*r*r*r+r2GsD      "  'r2c@seZdZdZed8ddZed8ddZed8dd Ze d d Z ed9ddZ ed:ddZ e d;dd(d)Zd?d+d,Zdddd-d.d@d4d5Zdddd-d.d@d6d7ZdS)AContextQAEvalChainz3LLM Chain for evaluating QA w/o GT based on contextrr8cCr9r:r*r;r*r*r+r=r>z%ContextQAEvalChain.is_lc_serializablecCr9)z.Whether the chain requires a reference string.Tr*r?r*r*r+rCz%ContextQAEvalChain.requires_referencecCr9)z+Whether the chain requires an input string.Tr*r?r*r*r+rDrz!ContextQAEvalChain.requires_inputr5r6rFr NonecCs0hd}|t|jkrtd|d|jdS)N>rIcontextrKrLrM)rOrPrQ)r<rFrRr*r*r+_validate_input_varssz'ContextQAEvalChain._validate_input_varsrcCr9)NzContextual Accuracyr*r?r*r*r+rAr>z"ContextQAEvalChain.evaluation_nameNrEr rGrHrcK&|pt}|||d||d|S)aLoad QA Eval Chain from LLM. Args: llm (BaseLanguageModel): the base language model to use. prompt (PromptTemplate): A prompt template containing the input_variables: 'query', 'context' and 'result' that will be used as the prompt for evaluation. Defaults to PROMPT. **kwargs: additional keyword arguments. Returns: ContextQAEvalChain: the loaded QA eval chain. rNNr*)r rr<rErFrHr*r*r+rSs zContextQAEvalChain.from_llmrIrrKrTrVr\rXrY context_keyr[rUrcr])r^cr_)rIrrKr*rarr[rXrYr*r+rerfz/ContextQAEvalChain.evaluate..rTrg)r@rVrXrYrr[rUrjr*rr+rk rlzContextQAEvalChain.evaluater-cCrmrnrorpr*r*r+rr"rsz"ContextQAEvalChain._prepare_outputFrtrxruryrvrwcKrz)Nrr{r|r}r*r*r+r~(s  z$ContextQAEvalChain._evaluate_stringscr)Nrrrr}r*r*r+r=rz%ContextQAEvalChain._aevaluate_stringsr)rFr rrrrn)rEr rFrGrHrrrr)rVr\rXr\rYrrrr[rrUrrr\rr)rrrrrr=rrCrDr rrrArSrkrrr~rr*r*r*r+rsF       rc@s>eZdZdZedddZedddZe ddddZd S)CotQAEvalChainz=LLM Chain for evaluating QA using chain of thought reasoning.rr8cCr9r:r*r;r*r*r+r=Rr>z!CotQAEvalChain.is_lc_serializablercCr9)NzCOT Contextual Accuracyr*r?r*r*r+rAVr>zCotQAEvalChain.evaluation_nameNrEr rFrGrHrcKr)zLoad QA Eval Chain from LLM.rNNr*)rrrr*r*r+rSZs zCotQAEvalChain.from_llmrrrn)rEr rFrGrHrrr) rrrrrr=rrArSr*r*r*r+rOs  r)rrrr)rrrr-)#r __future__rrr%typingrrrrrZ langchain_core.callbacks.managerrZlangchain_core.language_modelsr Zlangchain_core.promptsr Zpydanticr Zlangchain.chains.llmr Z#langchain.evaluation.qa.eval_promptr rrZlangchain.evaluation.schemarrZlangchain.schemarr,r1r2rrr*r*r*r+s&