o .if%@sfUdZddlmZmZmZmZmZmZmZddl m Z ddl m Z ddl mZddlmZddlmZddlmZdd lmZmZdd lmZmZdd lmZdd lmZmZdd l m!Z!ddl"m#Z#ddl$m%Z%m&Z&m'Z'ddl(m)Z)ddl*m+Z+m,Z,m-Z-ddl.m/Z/m0Z0ddl1m2Z2m3Z3de4deefddZ5ie+j6e'e+j7e&e+j8e%e+j9ee+j:e0e+j;ee+jee+j?ee+j@e3e+jAe2e+jBee+jCee+jDee+jEee+jFe!e+jGe#e+jHe)e+jIeiZJee+eee,eeee-ffeKd<ddde+dee dedeee-ffddZLddd d!ee+dee d"eeMdedeeee-ff d#d$ZNdS)%z Loading datasets and evaluators.)AnyDictListOptionalSequenceTypeUnion) ChatOpenAI)BaseLanguageModel)Chain)TrajectoryEvalChain)PairwiseStringEvalChain)LabeledPairwiseStringEvalChain)CriteriaEvalChainLabeledCriteriaEvalChain)EmbeddingDistanceEvalChain"PairwiseEmbeddingDistanceEvalChain)ExactMatchStringEvaluator)JsonEqualityEvaluatorJsonValidityEvaluator)JsonEditDistanceEvaluator)JsonSchemaEvaluator)ContextQAEvalChainCotQAEvalChain QAEvalChain)RegexMatchStringEvaluator) EvaluatorType LLMEvalChainStringEvaluator)LabeledScoreStringEvalChainScoreStringEvalChain)PairwiseStringDistanceEvalChainStringDistanceEvalChainurireturncCsHzddlm}Wn tytdw|d|}dd|dDS)aLoad a dataset from the `LangChainDatasets on HuggingFace `_. Args: uri: The uri of the dataset to load. Returns: A list of dictionaries, each representing a row in the dataset. **Prerequisites** .. code-block:: shell pip install datasets Examples -------- .. code-block:: python from langchain.evaluation import load_dataset ds = load_dataset("llm-math") r) load_datasetzXload_dataset requires the `datasets` package. Please install with `pip install datasets`zLangChainDatasets/cSsg|]}|qSr&).0dr&r&U/var/www/html/corbot_env/lib/python3.10/site-packages/langchain/evaluation/loading.py Fsz load_dataset..train)datasetsr% ImportError)r#r%datasetr&r&r)r%'s r%_EVALUATOR_MAPN)llm evaluatorr0kwargsc Ks|tvrtd|dttt|}t|trHz |p&tdddidd}Wnty=}z td|d |d }~ww|jd d |i|S|d i|S) a<Load the requested evaluation chain specified by a string. Parameters ---------- evaluator : EvaluatorType The type of evaluator to load. llm : BaseLanguageModel, optional The language model to use for evaluation, by default None **kwargs : Any Additional keyword arguments to pass to the evaluator. Returns ------- Chain The loaded evaluation chain. Examples -------- >>> from langchain.evaluation import load_evaluator, EvaluatorType >>> evaluator = load_evaluator(EvaluatorType.QA) zUnknown evaluator type: z Valid types are: zgpt-4seed*r)model model_kwargs temperaturezEvaluation with the z requires a language model to function. Failed to create the default 'gpt-4' model. Please manually provide an evaluation LLM or check your openai credentials.Nr0r&) r/ ValueErrorlistkeys issubclassrr Exceptionfrom_llm)r1r0r2 evaluator_clser&r&r)load_evaluatorcs,     r@)r0config evaluatorsrAcKsHg}|D]}|r||ini}|t|fd|ii||q|S)aeLoad evaluators specified by a list of evaluator types. Parameters ---------- evaluators : Sequence[EvaluatorType] The list of evaluator types to load. llm : BaseLanguageModel, optional The language model to use for evaluation, if none is provided, a default ChatOpenAI gpt-4 model will be used. config : dict, optional A dictionary mapping evaluator types to additional keyword arguments, by default None **kwargs : Any Additional keyword arguments to pass to all evaluators. Returns ------- List[Chain] The loaded evaluators. Examples -------- >>> from langchain.evaluation import load_evaluators, EvaluatorType >>> evaluators = [EvaluatorType.QA, EvaluatorType.CRITERIA] >>> loaded_evaluators = load_evaluators(evaluators, criteria="helpfulness") r0)getappendr@)rBr0rAr2loadedr1_kwargsr&r&r)load_evaluatorss !$rG)O__doc__typingrrrrrrr&langchain_community.chat_models.openair langchain_core.language_modelsr langchain.chains.baser 1langchain.evaluation.agents.trajectory_eval_chainr langchain.evaluation.comparisonr *langchain.evaluation.comparison.eval_chainr(langchain.evaluation.criteria.eval_chainrr,langchain.evaluation.embedding_distance.baserr%langchain.evaluation.exact_match.baser!langchain.evaluation.parsing.baserr*langchain.evaluation.parsing.json_distancer(langchain.evaluation.parsing.json_schemarlangchain.evaluation.qarrr%langchain.evaluation.regex_match.baserlangchain.evaluation.schemarrr'langchain.evaluation.scoring.eval_chainrr )langchain.evaluation.string_distance.baser!r"strr%QACOT_QA CONTEXT_QAPAIRWISE_STRING SCORE_STRINGLABELED_PAIRWISE_STRINGLABELED_SCORE_STRINGAGENT_TRAJECTORYCRITERIALABELED_CRITERIASTRING_DISTANCEPAIRWISE_STRING_DISTANCEEMBEDDING_DISTANCEPAIRWISE_EMBEDDING_DISTANCE JSON_VALIDITY JSON_EQUALITYJSON_EDIT_DISTANCEJSON_SCHEMA_VALIDATION REGEX_MATCH EXACT_MATCHr/__annotations__r@dictrGr&r&r&r)s$          $       6