o Zh!@sdZddlZddlmZmZddlmZmZmZm Z m Z m Z ddl m Z ddlmZerHddlmZddlmZdd lmZdd lmZmZmZeeZGd d d eZGd ddeZGdddeZdS)zQLoader that uses Playwright to load a page, then uses unstructured to parse html.N)ABCabstractmethod) TYPE_CHECKING AsyncIteratorDictIteratorListOptional)Document) BaseLoader)Browser)Page)Response)r r rc @sLeZdZdZedddddddefd d Zedd dd dd defddZdS)PlaywrightEvaluatorzAbstract base class for all evaluators. Each evaluator should take a page, a browser instance, and a response object, process the page as necessary, and return the resulting text. pager browserr responserreturncCsdS)aSynchronously process the page and return the resulting text. Args: page: The page to process. browser: The browser instance. response: The response from page.goto(). Returns: text: The text content of the page. Nselfrrrrrj/var/www/html/lang_env/lib/python3.10/site-packages/langchain_community/document_loaders/url_playwright.pyevaluates zPlaywrightEvaluator.evaluate AsyncPage AsyncBrowser AsyncResponsecsdS)aAsynchronously process the page and return the resulting text. Args: page: The page to process. browser: The browser instance. response: The response from page.goto(). Returns: text: The text content of the page. Nrrrrrevaluate_async*sz"PlaywrightEvaluator.evaluate_asyncN)__name__ __module__ __qualname____doc__rstrrrrrrrrs rc@s\eZdZdZddeeefddZdddd d d d efd dZddddd dd efddZ dS)UnstructuredHtmlEvaluatorz@Evaluate the page HTML content using the `unstructured` library.Nremove_selectorscCs.zddl}Wn tytdw||_dS)z%Initialize UnstructuredHtmlEvaluator.rNzQunstructured package not found, please install it with `pip install unstructured`) unstructured ImportErrorr#)rr#r$rrr__init__>s   z"UnstructuredHtmlEvaluator.__init__rr rr rrrc Cslddlm}|jp gD]}||}|D] }|r!|dqq |}||d}ddd|DS)z3Synchronously process the HTML content of the page.rpartition_htmlelement => element.remove()text cSg|]}t|qSrr!.0elrrr Vz6UnstructuredHtmlEvaluator.evaluate.. Zunstructured.partition.htmlr(r#locatorallZ is_visiblercontentjoin rrrrr(selectorelementselementZ page_sourcerrrrJs   z"UnstructuredHtmlEvaluator.evaluaterrrc sddlm}|jp gD] }||IdH}|D]}|IdHr+|dIdHqq |IdH}||d}ddd|DS) z4Asynchronously process the HTML content of the page.rr'Nr)r*r,cSr-rr.r/rrrr2fr3z.r4r9rrrrXs  z(UnstructuredHtmlEvaluator.evaluate_asyncN) rrrr r rr!r&rrrrrrr";s r"c@seZdZdZ     ddeedededeeedeed ee eeff d d Z d e e fd dZ d ee fddZd ee fddZdS)PlaywrightURLLoaderaLoad `HTML` pages with `Playwright` and parse with `Unstructured`. This is useful for loading pages that require javascript to render. Attributes: urls (List[str]): List of URLs to load. continue_on_failure (bool): If True, continue loading other URLs on failure. headless (bool): If True, the browser will run in headless mode. proxy (Optional[Dict[str, str]]): If set, the browser will access URLs through the specified proxy. Example: .. code-block:: python from langchain_community.document_loaders import PlaywrightURLLoader urls = ["https://api.ipify.org/?format=json",] proxy={ "server": "https://xx.xx.xx:15818", # https://: "username": "username", "password": "password" } loader = PlaywrightURLLoader(urls, proxy=proxy) data = loader.load() TNurlscontinue_on_failureheadlessr# evaluatorproxycCs^zddl}Wn tytdw||_||_||_||_|r&|r&td|p+t||_dS)z%Load a list of URLs using Playwright.rNzMplaywright package not found, please install it with `pip install playwright`z:`remove_selectors` and `evaluator` cannot be both not None) playwrightr%r?r@rArC ValueErrorr"rB)rr?r@rAr#rBrCrDrrrr&s   zPlaywrightURLLoader.__init__rc csddlm}|n}|jj|j|jd}|jD]S}z.|}||}|dur/t d|| d|j |||}d|i}t ||dVWqtyk} z|jr_td |d | n| WYd} ~ qd} ~ ww|WddS1s{wYdS) zLoad the specified URLs using Playwright and create Document instances. Returns: A list of Document instances with loaded content. r)sync_playwrightrArCN"page.goto() returned None for url loadsourceZ page_contentmetadataError fetching or processing , exception: )playwright.sync_apirFchromiumlaunchrArCr?new_pagegotorEwait_for_load_staterBrr Exceptionr@loggererrorclose) rrFprurlrrr+rLerrr lazy_loads2      "zPlaywrightURLLoader.lazy_loadcsdd|2IdHS)Load the specified URLs with Playwright and create Documents asynchronously. Use this function when in a jupyter notebook environment. Returns: A list of Document instances with loaded content. csg|z3dHW}|q6Sr=r)r0docrrrr2sz-PlaywrightURLLoader.aload..N) alazy_load)rrrraloadszPlaywrightURLLoader.aloadc Cs<ddlm}|4IdH}|jj|j|jdIdH}|jD]_}z:|IdH}||IdH}|duris.   r>)r loggingabcrrtypingrrrrrr Zlangchain_core.documentsr Z)langchain_community.document_loaders.baser rbr rr rrrrO getLoggerrrVrr"r>rrrrs       &.