o Zh@sfddlZddlmZmZmZmZmZddlmZddl m Z ddl m Z e eZGddde ZdS)N)AnyIteratorListOptionalSequence)Document) BaseLoader) NewsURLLoaderc @seZdZdZ    ddeeedeededed ed df d d Z d e e fd dZ e d eefddZd ee fddZdS) RSSFeedLoaderaQLoad news articles from `RSS` feeds using `Unstructured`. Args: urls: URLs for RSS feeds to load. Each articles in the feed is loaded into its own document. opml: OPML file to load feed urls from. Only one of urls or opml should be provided. The value can be a URL string, or OPML markup contents as byte or string. continue_on_failure: If True, continue loading documents even if loading fails for a particular URL. show_progress_bar: If True, use tqdm to show a loading progress bar. Requires tqdm to be installed, ``pip install tqdm``. **newsloader_kwargs: Any additional named arguments to pass to NewsURLLoader. Example: .. code-block:: python from langchain_community.document_loaders import RSSFeedLoader loader = RSSFeedLoader( urls=["", ""], ) docs = loader.load() The loader uses feedparser to parse RSS feeds. The feedparser library is not installed by default so you should install it if using this loader: https://pythonhosted.org/feedparser/ If you use OPML, you should also install listparser: https://pythonhosted.org/listparser/ Finally, newspaper is used to process each article: https://newspaper.readthedocs.io/en/latest/ NTFurlsopmlcontinue_on_failureshow_progress_barnewsloader_kwargsreturncKs:|du|dukr td||_||_||_||_||_dS)zInitialize with urls or OPML.Nz;Provide either the urls or the opml argument, but not both.) ValueErrorr r r rr)selfr r r rrr_/var/www/html/lang_env/lib/python3.10/site-packages/langchain_community/document_loaders/rss.py__init__/s  zRSSFeedLoader.__init__c CsR|}|jr%zddlm}Wnty }ztd|d}~ww||}t|S)Nr)tqdmzPackage tqdm must be installed if show_progress_bar=True. Please install with 'pip install tqdm' or set show_progress_bar=False.) lazy_loadrr ImportErrorlist)riterrerrrloadDszRSSFeedLoader.loadc CsX|jr|jSzddl}Wnty}ztd|d}~ww||j}dd|jDS)NrzPackage listparser must be installed if the opml arg is used. Please install with 'pip install listparser' or use the urls arg instead.cSsg|]}|jqSr)url).0feedrrr _sz+RSSFeedLoader._get_urls..)r listparserrparser Zfeeds)rr!rZrssrrr _get_urlsRs  zRSSFeedLoader._get_urlsc cs6zddl}Wn tytdw|jD]}z||}t|ddr/td|d|jWn"tyR}z|jrLt d|d|WYd}~q|d}~wwz"|j D]}t d d|j gi|j}|d}||jd<|VqWWqty}z|jrt d |j d|WYd}~q|d}~wwdS) NrzMfeedparser package not found, please install it with `pip install feedparser`ZbozoFzError fetching z , exception: r rzError processing entry r) feedparserrr#r"getattrrZbozo_exception Exceptionr loggererrorentriesr linkrrmetadata)rr$rrrentryloaderZarticlerrrrasT          zRSSFeedLoader.lazy_load)NNTF)__name__ __module__ __qualname____doc__rrstrboolrrrrrpropertyr#rrrrrrr s.$  r )loggingtypingrrrrrZlangchain_core.documentsrZ)langchain_community.document_loaders.baserZ)langchain_community.document_loaders.newsr getLoggerr.r'r rrrrs