o Zhs)@sZddlmZddlZddlmZmZddlZddl m Z m Z ddl m ZddlmZddlmZmZddlmZdd lmZmZmZdd lmZdd lmZdd lm Z dd l!m"Z"m#Z#ddl$m%Z%ddl&m'Z'm(Z(m)Z)ddl*m+Z+ddl,m-Z-ddl.m/Z/ddl0m1Z1m2Z2m3Z3m4Z4ej5rddl6m7Z7        d-d.d*d+Z8eGd,d)d)e9Z:dS)/) annotationsN) dataclassfield)Datasetconcatenate_datasets) Embeddings)BaseLanguageModel)EvaluationEventtrack) new_group)BaseRagasEmbeddingsLangchainEmbeddingsWrapperembedding_factory)ExceptionInRunner)Executor) llm_factory) BaseRagasLLMLangchainLLMWrapper)AnswerCorrectness)MetricMetricWithEmbeddings MetricWithLLM)AspectCritique) RunConfigget_feature_language)handle_deprecated_ground_truthsremap_column_namesvalidate_column_dtypesvalidate_evaluation_modes) CallbacksFTdatasetrmetricslist[Metric] | Nonellm't.Optional[BaseRagasLLM | LangchainLLM] embeddings5t.Optional[BaseRagasEmbeddings | LangchainEmbeddings] callbacksr is_asyncbool run_configt.Optional[RunConfig]raise_exceptions column_mapt.Optional[t.Dict[str, str]]returnResultc s|pi}|pg}|durtdpt|dur+ddlm} m} m} m} | | | | g}t||}t|}t ||t |t |t rHt |d}t |trQt|}g} g}g}d}t|D]M\}t |trl| |jt |tr|jdur|dur}t}||_|t |tr|jdur|durt}||_|t |tr|jdur}q]fdd|Dtd d |d g}td i|d \}}t|D]1\ttj t!tj"ftd|d \}||ffdd|Dqg}zzE#}|gkrt$t|D]2\}i}t|D]\}}|t%||||j<q|||\}j&sC|'|qWnt(y^}z |j&sX|)||d}~wwt*t+,||| d}|j&sr|'|W|D] dtt|_qu|D] dtt|_q|dkrdtt||_n-|D] dtt|_q|D] dtt|_q|dkrdtt||_wdd|D}dd|D}t-.dd|D}t/t0d|d|j1dt%|dkr|dndd|S)aQ Run the evaluation on the dataset with different metrics Parameters ---------- dataset : Dataset[question: list[str], contexts: list[list[str]], answer: list[str], ground_truth: list[list[str]]] The dataset in the format of ragas which the metrics will use to score the RAG pipeline with metrics : list[Metric] , optional List of metrics to use for evaluation. If not provided then ragas will run the evaluation on the best set of metrics to give a complete view. llm: BaseRagasLLM, optional The language model to use for the metrics. If not provided then ragas will use the default language model for metrics which require an LLM. This can we overridden by the llm specified in the metric level with `metric.llm`. embeddings: BaseRagasEmbeddings, optional The embeddings to use for the metrics. If not provided then ragas will use the default embeddings for metrics which require embeddings. This can we overridden by the embeddings specified in the metric level with `metric.embeddings`. callbacks: Callbacks, optional Lifecycle Langchain Callbacks to run during evaluation. Check the [langchain documentation](https://python.langchain.com/docs/modules/callbacks/) for more information. is_async: bool, optional Whether to run the evaluation in async mode or not. If set to True then the evaluation is run by calling the `metric.ascore` method. In case the llm or embeddings does not support async then the evaluation can be run in sync mode with `is_async=False`. Default is False. run_config: RunConfig, optional Configuration for runtime settings like timeout and retries. If not provided, default values are used. raise_exceptions: True Whether to raise exceptions or not. If set to True then the evaluation will raise an exception if any of the metrics fail. If set to False then the evaluation will return `np.nan` for the row that failed. Default is True. column_map : dict[str, str], optional The column names of the dataset to use for evaluation. If the column names of the dataset are different from the default ones then you can provide the mapping as a dictionary here. Example: If the dataset column name is contexts_v1, column_map can be given as {"contexts":"contexts_v1"} Returns ------- Result Result object containing the scores of each metric. You can use this do analysis later. Raises ------ ValueError if validation fails because the columns required for the metrics are missing or if the columns are of the wrong format. Examples -------- the basic usage is as follows: ``` from ragas import evaluate >>> dataset Dataset({ features: ['question', 'ground_truth', 'answer', 'contexts'], num_rows: 30 }) >>> result = evaluate(dataset) >>> print(result) {'context_precision': 0.817, 'faithfulness': 0.892, 'answer_relevancy': 0.874} ``` NzProvide dataset!r)answer_relevancycontext_precisioncontext_recall faithfulnessr+csg|]}|qS)init.0mr6r8G/var/www/html/lang_env/lib/python3.10/site-packages/ragas/evaluation.py szevaluate..Z EvaluatingT)ZdescZkeep_progress_barr-r+zragas evaluation)nameZinputsr(r)zrow c s.g|]}j|j|jddqS)-r?)ZsubmitZascorer?)r;metric)executorir)row row_group_cmr8r=r>s )scoresr!binary_columnscSsg|]}|jqSr8rAr:r8r8r=r>scSsg|]}t|qSr8rr:r8r8r=r>scSsg|]}|dur|qS)Nr8r:r8r8r=r>sZ evaluation)Z event_typer"Zevaluation_modeZnum_rowslanguage)2 ValueErrorrZ ragas.metricsr2r3r4r5rrrr isinstance LangchainLLMrLangchainEmbeddingsr enumeraterappendr?rr$rrr&rrZanswer_similarityrr tcastDictstrAnyresultsrlenZendedZ on_chain_end ExceptionZon_chain_errorr1r from_listnpuniquer r shape) r!r"r$r&r(r)r+r-r.r2r3r4r5Zbinary_metricsZ llm_changedZembeddings_changedZanswer_correctness_is_setrBZrow_run_managersZ evaluation_rmZevaluation_group_cmZrow_rmrGrV_sjr<eresultZ metrics_namesZ metric_langr8)rCrDr)rErFr+r=evaluate(sS                         rbc@sReZdZUded<dZded<eedZded<d d ZddddZ dddZ dS)r1rrGNzt.Optional[Dataset]r!)default_factoryz t.List[str]rHcCsXg}|jdD] }t|j|}|||<||jvr)tt|}||dq dS)Nrg|=) rGkeysrZZnanmeanrHrQrRfloatrP)selfvaluesZcnvaluer8r8r= __post_init__s  zResult.__post_init__F batch_size int | Nonebatchedr*cCsP|jdur td|jjd|jjdksJt|j|jgdd}|j||dS)Nz-dataset is not provided for the results classr)Zaxis)rjrl)r!rKrGr\r to_pandas)rfrjrlZ result_dsr8r8r=rns zResult.to_pandasr0rTcCs,|}dd|D}dd|dS)NcSs"g|] \}}d|d|dqS)'z': z0.4fr8)r;kvr8r8r=r>$s"z#Result.__repr__..{z, })copyitemsjoin)rfrGZ score_strsr8r8r=__repr__"szResult.__repr__)NF)rjrkrlr*)r0rT) __name__ __module__ __qualname____annotations__r!rlistrHrirnrwr8r8r8r=r1 s   )NNNNFNTN)r!rr"r#r$r%r&r'r(r r)r*r+r,r-r*r.r/r0r1); __future__rtypingrQ dataclassesrrnumpyrZZdatasetsrrZlangchain_core.embeddingsrrNZlangchain_core.language_modelsrrMZragas._analyticsr r Zragas.callbacksr Zragas.embeddings.baser r rZragas.exceptionsrZragas.executorrZ ragas.llmsrZragas.llms.baserrZ!ragas.metrics._answer_correctnessrZragas.metrics.baserrrZragas.metrics.critiquerZragas.run_configrZ ragas.utilsrZragas.validationrrrr TYPE_CHECKINGZlangchain_core.callbacksr rbdictr1r8r8r8r=sD             d