o /if,@s`ddlmZddlZddlmZmZddlZddl m Z m Z ddl m ZddlmZddlmZmZddlmZdd lmZmZmZdd lmZdd lmZdd lm Z dd l!m"Z"m#Z#ddl$m%Z%ddl&m'Z'm(Z(m)Z)m*Z*ddl+m,Z,ddl-m.Z.ddl/m0Z0ddl1m2Z2m3Z3m4Z4m5Z5ej6rddl7m8Z8         d.d/d+d,Z9eGd-d*d*e:Z;dS)0) annotationsN) dataclassfield)Datasetconcatenate_datasets) Embeddings)BaseLanguageModel)EvaluationEventtrack) new_group)BaseRagasEmbeddingsLangchainEmbeddingsWrapperembedding_factory)ExceptionInRunner)Executor) llm_factory) BaseRagasLLMLangchainLLMWrapper)AnswerCorrectness)MetricMetricWithEmbeddings MetricWithLLMis_reproducable)AspectCritique) RunConfigget_feature_language)handle_deprecated_ground_truthsremap_column_namesvalidate_column_dtypesvalidate_evaluation_modes) CallbacksFTdatasetrmetricslist[Metric] | Nonellm't.Optional[BaseRagasLLM | LangchainLLM] embeddings5t.Optional[BaseRagasEmbeddings | LangchainEmbeddings] callbacksr!in_ciboolis_async run_configt.Optional[RunConfig]raise_exceptions column_mapt.Optional[t.Dict[str, str]]returnResultc " sP| pi} |pg}|durtd|pt}|dur+ddlm} m} m} m} | | | | g}t|| }t|}t ||t |t |t rHt ||d}t |trQt|}g}g}g}g}d}t|D]e\}t |trn||jt |tr|jdur|durt}||_|t |tr|jdur|durt}||_|t |tr|jdur}|rt|r|jdkrd|_|||q_td d ||d g}t d i|d \}}t|D]1\t!"t!j#t$t!j%ft d|d \}||ffdd|Dqg}zzE&}|gkrt't|D]2\}i}t|D]\}}|t(||||j<q-|||\}j)sT|*|q#Wnt+yo}z |j)si|,||d}~wwt-t./|||d}|j)s|*|W|D] dt!"t|_q|D] dt!"t|_q|dkrdt!"t||_|D]d|_qn8|D] dt!"t|_q|D] dt!"t|_q|dkrdt!"t||_|D]d|_qwdd|D} dd|D}!t01dd|!D}!t2t3d| d|j4dt(|!dkr |!dnd|d|S)aJ Run the evaluation on the dataset with different metrics Parameters ---------- dataset : Dataset[question: list[str], contexts: list[list[str]], answer: list[str], ground_truth: list[list[str]]] The dataset in the format of ragas which the metrics will use to score the RAG pipeline with metrics : list[Metric] , optional List of metrics to use for evaluation. If not provided then ragas will run the evaluation on the best set of metrics to give a complete view. llm: BaseRagasLLM, optional The language model to use for the metrics. If not provided then ragas will use the default language model for metrics which require an LLM. This can we overridden by the llm specified in the metric level with `metric.llm`. embeddings: BaseRagasEmbeddings, optional The embeddings to use for the metrics. If not provided then ragas will use the default embeddings for metrics which require embeddings. This can we overridden by the embeddings specified in the metric level with `metric.embeddings`. callbacks: Callbacks, optional Lifecycle Langchain Callbacks to run during evaluation. Check the [langchain documentation](https://python.langchain.com/docs/modules/callbacks/) for more information. in_ci: bool Whether the evaluation is running in CI or not. If set to True then some metrics will be run to increase the reproducability of the evaluations. This will increase the runtime and cost of evaluations. Default is False. is_async: bool Whether to run the evaluation in async mode or not. If set to True then the evaluation is run by calling the `metric.ascore` method. In case the llm or embeddings does not support async then the evaluation can be run in sync mode with `is_async=False`. Default is False. run_config: RunConfig, optional Configuration for runtime settings like timeout and retries. If not provided, default values are used. raise_exceptions: True Whether to raise exceptions or not. If set to True then the evaluation will raise an exception if any of the metrics fail. If set to False then the evaluation will return `np.nan` for the row that failed. Default is True. column_map : dict[str, str], optional The column names of the dataset to use for evaluation. If the column names of the dataset are different from the default ones then you can provide the mapping as a dictionary here. Example: If the dataset column name is contexts_v1, column_map can be given as {"contexts":"contexts_v1"} Returns ------- Result Result object containing the scores of each metric. You can use this do analysis later. Raises ------ ValueError if validation fails because the columns required for the metrics are missing or if the columns are of the wrong format. Examples -------- the basic usage is as follows: ``` from ragas import evaluate >>> dataset Dataset({ features: ['question', 'ground_truth', 'answer', 'contexts'], num_rows: 30 }) >>> result = evaluate(dataset) >>> print(result) {'context_precision': 0.817, 'faithfulness': 0.892, 'answer_relevancy': 0.874} ``` NzProvide dataset!r)answer_relevancycontext_precisioncontext_recall faithfulness)r- EvaluatingT)desckeep_progress_barr/r-zragas evaluation)nameinputsr)r,zrow c s.g|]}j|j|jddqS)-r>)submitascorer>).0metricexecutorir,row row_group_cmI/var/www/html/corbot_env/lib/python3.10/site-packages/ragas/evaluation.py s zevaluate..)scoresr"binary_columnscSsg|]}|jqSrKrArDmrKrKrLrMscSsg|]}t|qSrKrrPrKrKrLrMscSsg|]}|dur|qS)NrKrPrKrKrLrMs evaluation) event_typer#evaluation_modenum_rowslanguager*)5 ValueErrorr ragas.metricsr4r5r6r7rrr r isinstance LangchainLLMrLangchainEmbeddingsr enumeraterappendr>rr%rrr'rranswer_similarityrreproducibilityinitrr tcastDictstrAnyresultsrlenended on_chain_end Exceptionon_chain_errorr3r from_listnpuniquer r shape)"r"r#r%r'r)r*r,r-r/r0r4r5r6r7binary_metrics llm_changedembeddings_changedreproducable_metricsanswer_correctness_is_setrErow_run_managers evaluation_rmevaluation_group_cmrow_rmrNrg_sjrQeresult metrics_names metric_langrKrFrLevaluate-s X                             rc@sReZdZUded<dZded<eedZded<d d ZddddZ dddZ dS)r3rrNNzt.Optional[Dataset]r")default_factoryz t.List[str]rOcCsXg}|jdD] }t|j|}|||<||jvr)tt|}||dq dS)Nrg|=) rNkeysrnnanmeanrOrbrcfloatr^)selfvaluescnvaluerKrKrL __post_init__(s  zResult.__post_init__F batch_size int | Nonebatchedr+cCsP|jdur td|jjd|jjdksJt|j|jgdd}|j||dS)Nz-dataset is not provided for the results classrr9)axis)rr)r"rXrNrpr to_pandas)rrr result_dsrKrKrLr1s zResult.to_pandasr2recCs,|}dd|D}dd|dS)NcSs"g|] \}}d|d|dqS)'z': z0.4frK)rDkvrKrKrLrM;s"z#Result.__repr__..{z, })copyitemsjoin)rrN score_strsrKrKrL__repr__9szResult.__repr__)NF)rrrr+)r2re) __name__ __module__ __qualname____annotations__r"rlistrOrrrrKrKrKrLr3"s   ) NNNNFTNTN)r"rr#r$r%r&r'r(r)r!r*r+r,r+r-r.r/r+r0r1r2r3)< __future__rtypingrb dataclassesrrnumpyrndatasetsrrlangchain_core.embeddingsrr\langchain_core.language_modelsrr[ragas._analyticsr r ragas.callbacksr ragas.embeddings.baser r rragas.exceptionsrragas.executorr ragas.llmsrragas.llms.baserr!ragas.metrics._answer_correctnessrragas.metrics.baserrrrragas.metrics.critiquerragas.run_configr ragas.utilsrragas.validationrrrr TYPE_CHECKINGlangchain_core.callbacksr!rdictr3rKrKrKrLsF             v