o Zh2@sTddlZddlmZmZmZddlmZddlmZddl m Z GdddeZ dS)N)IteratorLiteralOptional) BaseLoader)Document) get_from_envc @seZdZdZdedefddZdedefddZddd dd d ed eed eede ddeef ddZ de e fddZ dS)FireCrawlLoadera! FireCrawlLoader document loader integration Setup: Install ``firecrawl-py``,``langchain_community`` and set environment variable ``FIRECRAWL_API_KEY``. .. code-block:: bash pip install -U firecrawl-py langchain_community export FIRECRAWL_API_KEY="your-api-key" Instantiate: .. code-block:: python from langchain_community.document_loaders import FireCrawlLoader loader = FireCrawlLoader( url = "https://firecrawl.dev", mode = "crawl" # other params = ... ) Lazy load: .. code-block:: python docs = [] docs_lazy = loader.lazy_load() # async variant: # docs_lazy = await loader.alazy_load() for doc in docs_lazy: docs.append(doc) print(docs[0].page_content[:100]) print(docs[0].metadata) .. code-block:: python Introducing [Smart Crawl!](https://www.firecrawl.dev/smart-crawl) Join the waitlist to turn any web {'ogUrl': 'https://www.firecrawl.dev/', 'title': 'Home - Firecrawl', 'robots': 'follow, index', 'ogImage': 'https://www.firecrawl.dev/og.png?123', 'ogTitle': 'Firecrawl', 'sitemap': {'lastmod': '2024-08-12T00:28:16.681Z', 'changefreq': 'weekly'}, 'keywords': 'Firecrawl,Markdown,Data,Mendable,Langchain', 'sourceURL': 'https://www.firecrawl.dev/', 'ogSiteName': 'Firecrawl', 'description': 'Firecrawl crawls and converts any website into clean markdown.', 'ogDescription': 'Turn any website into LLM-ready data.', 'pageStatusCode': 200, 'ogLocaleAlternate': []} Async load: .. code-block:: python docs = await loader.aload() print(docs[0].page_content[:100]) print(docs[0].metadata) .. code-block:: python Introducing [Smart Crawl!](https://www.firecrawl.dev/smart-crawl) Join the waitlist to turn any web {'ogUrl': 'https://www.firecrawl.dev/', 'title': 'Home - Firecrawl', 'robots': 'follow, index', 'ogImage': 'https://www.firecrawl.dev/og.png?123', 'ogTitle': 'Firecrawl', 'sitemap': {'lastmod': '2024-08-12T00:28:16.681Z', 'changefreq': 'weekly'}, 'keywords': 'Firecrawl,Markdown,Data,Mendable,Langchain', 'sourceURL': 'https://www.firecrawl.dev/', 'ogSiteName': 'Firecrawl', 'description': 'Firecrawl crawls and converts any website into clean markdown.', 'ogDescription': 'Turn any website into LLM-ready data.', 'pageStatusCode': 200, 'ogLocaleAlternate': []} paramsreturncCsd}gd}|D] }||rd}nq|rtdtd|vr/|ddur,|d|d<|d=d|vrB|ddur?|d|d<|d=d |vrU|d durR|d |d <|d =d |vrh|d dure|d |d <|d =d |vrt|d tr|||d |d<|d =|S)NF)includesexcludesallowBackwardCrawlingallowExternalContentLinks pageOptionsTBDeprecated parameters detected. See Firecrawl v1 docs for updates.r Z includePathsr Z excludePathsr ZallowBackwardLinksrZallowExternalLinksrZ scrapeOptions)getwarningswarnDeprecationWarning isinstancedictlegacy_scrape_options_adapter)selfr use_legacy_optionsZ legacy_keyskeyre/var/www/html/lang_env/lib/python3.10/site-packages/langchain_community/document_loaders/firecrawl.pylegacy_crawler_options_adapterCsF         z.FireCrawlLoader.legacy_crawler_options_adaptercCsd}dg}d|vrsd|dvrs|dddks'|dddks'|dddkrsd}d |dvrH|dd r>|dd |d <n |dd d |d <d |dvr\|dd r\|dd |d <d|dvrp|ddrp|dd|d <|d=gd}|D] }||rd}nqy|r8tdtd|vr|ddur|d|d=d|vr|ddur|d|d=d|vr|ddur|d|d=d|vr|ddur|d|d=d|vr|ddur|d|d=d|vr|ddur|d|d=d|vr|ddur |d|d=d|vr#|ddur |d|d<|d=d|vr8|ddur5|d|d <|d=d!|vrA||d!<|S)"NFmarkdownZextractorOptionsmodezllm-extractionzllm-extraction-from-raw-htmlzllm-extraction-from-markdownTZextractionPromptpromptz-Extract page information based on the schema.ZextractionSchemaZschemaZ userPrompt) includeMarkdown includeHtmlincludeRawHtmlincludeExtract includeLinks screenshotfullPageScreenshotonlyIncludeTags removeTagsrr!r"htmlr#rawHtmlr$extractr%linksr&r'zscreenshot@fullPager(Z includeTagsr)Z excludeTagsformats)rrrrremoveappend)rr rr.Z scrape_keysrrrrrss                              z-FireCrawlLoader.legacy_scrape_options_adapterNcrawl)api_keyapi_urlrr urlr2r3r)r1scrapemapr,cCszddlm}Wn tytdw|dvr td|d|s&td|p,tdd }|||d |_||_||_|p=i|_d S) aInitialize with API key and url. Args: url: The url to be crawled. api_key: The Firecrawl API key. If not specified will be read from env var FIRECRAWL_API_KEY. Get an API key api_url: The Firecrawl API URL. If not specified will be read from env var FIRECRAWL_API_URL or defaults to https://api.firecrawl.dev. mode: The mode to run the loader in. Default is "crawl". Options include "scrape" (single url), "crawl" (all accessible sub pages), "map" (returns list of links that are semantically related). "extract" (extracts structured data from a page). params: The parameters to pass to the Firecrawl API. Examples include crawlerOptions. For more details, visit: https://github.com/mendableai/firecrawl-py r) FirecrawlAppzD`firecrawl` package not found, please run `pip install firecrawl-py`)r1r5searchr6r,Invalid mode 'zJ'. Allowed: 'crawl', 'scrape', 'search', 'map', 'extract'.zUrl must be providedr2ZFIRECRAWL_API_KEY)r2r3N) firecrawlr7 ImportError ValueErrorrr4rr )rr4r2r3rr r7rrr__init__s$  zFireCrawlLoader.__init__ccs^|jdkr|jj|j||jdg}nd|jdkr5|js!td|jj|j||jd}| dg}nD|jdkrL|jsAtd|jj |j|jd}n-|jdkrg|jsXtd t |jj |jg|jdg}n|jd krptd td |jd |D]1}|jdks|jdkr|}i}n| dp| dp| dd}| di}|sq{t ||dVq{dS)Nr5)r r1zURL is required for crawl modedatar6zURL is required for map moder,z URL is required for extract moder8z?Search mode is not supported in this version, please downgrade.r9z@'. Allowed: 'crawl', 'scrape', 'map', 'extract'.rr*r+metadata) page_contentr@)rr:Z scrape_urlr4rr r<Z crawl_urlrrZmap_urlstrr,r)rZfirecrawl_docsZcrawl_responsedocrAr@rrr lazy_loadsV        zFireCrawlLoader.lazy_load)__name__ __module__ __qualname____doc__rrrrBrrr=rrrDrrrrr s(90p 0r) rtypingrrrZlangchain_core.document_loadersrZlangchain_core.documentsrZlangchain_core.utilsrrrrrrs