o Zh@sVdZddlZddlmZmZmZddlmZddlm Z e e Z Gddde Z dS)z1Loader that uses unstructured to load HTML files.N)AnyIteratorList)Document) BaseLoaderc@sheZdZdZ    ddeededededed ed d fd d Zd ee fddZ d e e fddZ d S) NewsURLLoadera/Load news articles from URLs using `Unstructured`. Args: urls: URLs to load. Each is loaded into its own document. text_mode: If True, extract text from URL and use that for page content. Otherwise, extract raw HTML. nlp: If True, perform NLP on the extracted contents, like providing a summary and extracting keywords. continue_on_failure: If True, continue loading documents even if loading fails for a particular URL. show_progress_bar: If True, use tqdm to show a loading progress bar. Requires tqdm to be installed, ``pip install tqdm``. **newspaper_kwargs: Any additional named arguments to pass to newspaper.Article(). Example: .. code-block:: python from langchain_community.document_loaders import NewsURLLoader loader = NewsURLLoader( urls=["", ""], ) docs = loader.load() Newspaper reference: https://newspaper.readthedocs.io/en/latest/ TFurls text_modenlpcontinue_on_failureshow_progress_barnewspaper_kwargsreturnNcKsTz ddl}|j|_Wn tytdw||_||_||_||_||_||_ dS)zInitialize with file path.rNzMnewspaper package not found, please install it with `pip install newspaper3k`) newspaper __version__Z_NewsURLLoader__version ImportErrorrr r r r r )selfrr r r r r rr`/var/www/html/lang_env/lib/python3.10/site-packages/langchain_community/document_loaders/news.py__init__+s    zNewsURLLoader.__init__c CsR|}|jr%zddlm}Wnty }ztd|d}~ww||}t|S)Nr)tqdmzPackage tqdm must be installed if show_progress_bar=True. Please install with 'pip install tqdm' or set show_progress_bar=False.) lazy_loadr rrlist)riterrerrrloadFszNewsURLLoader.loadc csRzddlm}Wnty}ztd|d}~ww|jD]}z||fi|j}|||jr9|Wn"ty\}z|j rVt d|d|WYd}~q|d}~wwt |ddt |dt |d dt |d gt |d dt |d dt |d dd}|j r|j}n|j}|jrt |dg|d<t |dd|d<t||dVqdS)Nr)ArticlezFCannot import newspaper, please install with `pip install newspaper3k`zError fetching or processing z , exception: titleurlZcanonical_linkauthorsZ meta_langZmeta_description publish_date)rlinkr language descriptionr!keywordssummary)Z page_contentmetadata)rrrrr downloadparser Exceptionr loggererrorgetattrr texthtmlr)rrrrZarticler'contentrrrrTsR        zNewsURLLoader.lazy_load)TFTF) __name__ __module__ __qualname____doc__rstrboolrrrrrrrrrrr s.  r)r4loggingtypingrrrZlangchain_core.documentsrZ)langchain_community.document_loaders.baser getLoggerr1r+rrrrrs