o Zh@sdZddlmZddlZddlZddlZddlZddlZddlmZddl m Z ddl m Z ddl mZmZmZmZmZmZmZmZmZmZmZddlmZddlZddlZdd lmZdd lm Z dd l!m"Z"dd l#m$Z$m%Z%erddl&Z&ddl'Z'ddl(Z(ddl)Z)dd l*m+Z+gdZ,gdZ-d;ddZ.e/e0Z1dZ2dZ3dZ4dZ5hdZ6dd+d,Z;Gd-d.d.e ZGd3d4d4e Z?Gd5d6d6e Z@Gd7d8d8e ZAGd9d:d:e ZBdS)?z(Module contains common parsers for PDFs.) annotationsN)datetime)Path)TemporaryDirectory) TYPE_CHECKINGAnyBinaryIOIterableIteratorLiteralMappingOptionalSequenceUnioncast)urlparse)Document)BaseBlobParser)Blob)BaseImageBlobParserRapidOCRBlobParser)TextLinearizationConfig)Z DCTDecodeZDCTZ JPXDecode) Z LZWDecodeZLZWZ FlateDecodeZFlZ ASCII85DecodeZA85ZASCIIHexDecodeZAHxZRunLengthDecodeZRLZCCITTFaxDecodeZCCFZ JBIG2Decodeimages,Sequence[Union[Iterable[np.ndarray], bytes]]returnstrcCslzddlm}Wn tytdw|}d}|D]}||\}}|r3dd|D}|d|7}q|S)zExtract text from images with RapidOCR. Args: images: Images to extract text from. Returns: Text extracted from images. Raises: ImportError: If `rapidocr-onnxruntime` package is not installed. r)RapidOCRzc`rapidocr-onnxruntime` package not found, please install it with `pip install rapidocr-onnxruntime`cSsg|]}|dqS)).0textrrg/var/www/html/lang_env/lib/python3.10/site-packages/langchain_community/document_loaders/parsers/pdf.py Zz5extract_from_images_with_rapidocr.. )Zrapidocr_onnxruntimer ImportErrorjoin)rrZocrr!imgresult_rrr"!extract_from_images_with_rapidocr@s   r+z {image_text} r%z >producer total_pages creationdatesourcecreatorblobrcontentformatcCs`|r.|jpd}|dkr|dd}d|d|d}|S|dkr.d tj|d d d |d }|S)aFormat the content of the image with the source of the blob. blob: The blob containing the image. format:: The format for the parsed output. - "text" = return the content as is - "markdown-img" = wrap the content into an image markdown link, w/ link pointing to (`![body)(#)`] - "html-img" = wrap the content as the `alt` text of an tag and link to (`{body}`) #z markdown-img]z\\]z![z]()zhtml-imgz T)quotez src=)r/replacehtmlescape)r1r2r3r/rrr"_format_inner_imageis  r;metadatadict[str, Any]cCs4t|s tdt|ddtstd|S)zValidate that the metadata has all the standard keys and the page is an integer. The standard keys are: - source - total_page - creationdate - creator - producer Validate that page is an integer if it is present. z3The PDF parser must valorize the standard metadata.pagerz(The PDF metadata page must be a integer.)_STD_METADATA_KEYSissubsetkeys ValueError isinstancegetint)r<rrr"_validate_metadatas  rFc Csi}ddd}|D]f\}}t|ttfvrt|}|dr&|dd}|}|dvrMzt|dd d  d ||<Wq t yL|||<Yq w||vr\||||<|||<q t |trh| ||<q t |trq|||<q |S) zPurge metadata from unwanted keys and normalize key names. Args: metadata: The original metadata dictionary. Returns: The cleaned and normalized the key format of metadata dictionary. r-r/)Z page_count file_path/rN)r.Zmoddate'rzD:%Y%m%d%H%M%S%zT) itemstyperrE startswithlowerrstrptimer8 isoformatrBrCstrip)r<Z new_metadataZmap_keykvrrr"_purge_metadatas:          rTz  extras list[str]text_from_pagecsRdfd d ||d }|s'd }d tdd|}|r#td|}||}|S)a5Insert extras such as image/table in a text between two paragraphs if possible, else at the end of the text. Args: extras: List of extra content (images/tables) to insert. text_from_page: The text content from the page. Returns: The merged text with extras inserted. rVrWrXrrecursboolr Optional[str]c s|rPtD]G}||}|dkrKd}|r||d|d}|r(|||d}n d}dtdd|}|r:||}|d||||d}|Sqd}|S|}|S)NFrrUcS|SNrxrrr"zO_merge_text_and_extras.._recurs_merge_text_and_extras..)_PARAGRAPH_DELIMITERrfindr'filter) rVrXrYdelimposZ previous_textall_text all_extras str_extras_recurs_merge_text_and_extrasrr"rls0 z=_merge_text_and_extras.._recurs_merge_text_and_extrasTrrUcSr]r^rr_rrr"rarbz(_merge_text_and_extras..r\N)rVrWrXrrYrZrr[)r'rerc)rVrXrhrirjrrkr"_merge_text_and_extrass  rmcsLeZdZdZ  d#dedddddd$fddZd%ddZd&d!d"ZZS)' PyPDFParseraParse a blob from a PDF using `pypdf` library. This class provides methods to parse a blob from a PDF document, supporting various configurations such as handling password-protected PDFs, extracting images. It integrates the 'pypdf' library for PDF processing and offers synchronous blob parsing. Examples: Setup: .. code-block:: bash pip install -U langchain-community pypdf Load a blob from a PDF file: .. code-block:: python from langchain_core.documents.base import Blob blob = Blob.from_path("./example_data/layout-parser-paper.pdf") Instantiate the parser: .. code-block:: python from langchain_community.document_loaders.parsers import PyPDFParser parser = PyPDFParser( # password = None, mode = "single", pages_delimiter = " ", # images_parser = TesseractBlobParser(), ) Lazily parse the blob: .. code-block:: python docs = [] docs_lazy = parser.lazy_parse(blob) for doc in docs_lazy: docs.append(doc) print(docs[0].page_content[:100]) print(docs[0].metadata) NFr>r!plain)modepages_delimiter images_parserimages_inner_formatextraction_modeextraction_kwargspasswordOptional[Union[str, bytes]]extract_imagesrZrpLiteral['single', 'page']rqrrrOptional[BaseImageBlobParser]rs+Literal['text', 'markdown-img', 'html-img']rtLiteral['plain', 'layout']ruOptional[dict[str, Any]]c s`t|dvr td||_|r|st}||_||_||_||_||_ ||_ |p,i|_ dS)uInitialize a parser based on PyPDF. Args: password: Optional password for opening encrypted PDFs. extract_images: Whether to extract images from the PDF. mode: The extraction mode, either "single" for the entire document or "page" for page-wise extraction. pages_delimiter: A string delimiter to separate pages in single-mode extraction. images_parser: Optional image blob parser. images_inner_format: The format for the parsed output. - "text" = return the content as is - "markdown-img" = wrap the content into an image markdown link, w/ link pointing to (`![body)(#)`] - "html-img" = wrap the content as the `alt` text of an tag and link to (`{body}`) extraction_mode: “plain” for legacy functionality, “layout” extract text in a fixed width format that closely adheres to the rendered layout in the source pdf. extraction_kwargs: Optional additional parameters for the extraction process. Raises: ValueError: If the `mode` is not "single" or "page". singler>mode must be single or pageN) super__init__rBrxrrrrsrvrprqrtru) selfrvrxrprqrrrsrtru __class__rr"r$s %zPyPDFParser.__init__r1rrIterator[Document]c #sNzddlWn tytdwdfdd }|~}j|jd }td d d d tt|jp4iB|j t |j dB}g}t |j D]2\}}||d}  |} t| g| } jdkrut| t|||j|dBdVqH|| qHjdkrtj|t|dVWddSWddS1swYdS)m Lazily parse the blob. Insert image, if possible, between two paragraphs. In this way, a paragraph can be continued on the next page. Args: blob: The blob to parse. Raises: ImportError: If the `pypdf` package is not found. Yield: An iterator over the parsed documents. rNzE`pypdf` package not found, please install it with `pip install pypdf`r>pypdf.PageObjectrrcs,jdr |S|jddjijS)z Extract text from image given the version of pypdf. Args: page: The page object to extract text from. Returns: str: The extracted text. 3rtNr) __version__rM extract_textrtrur>pypdfrrr"_extract_text_from_pagems z7PyPDFParser.lazy_parse.._extract_text_from_pagervZPyPDFr)r,r0r.)r/r-r)r>Z page_label page_contentr<r)r>rrr)rr& as_bytes_ioZ PdfReaderrvrTrdictr<r/lenpages enumerateextract_images_from_pagermrQrprrFZ page_labelsappendrqr') rr1r pdf_file_obj pdf_reader doc_metadataZ single_texts page_numberr>rXimages_from_pagerhrrr" lazy_parseWsb          "zPyPDFParser.lazy_parsepypdf._page.PageObjectc Csn|jsdSddlm}dtt|dvrdS|dd}g}|D]}d}||ddkr||d d dtvr[||d ||d }}tj || tj d  ||d}n!||d d dt vrwt|t|| }ntd|durt} ||j| ddtj| dd} t|j| j} |t| | |jq$tj t!"t#d|dS)Extract images from a PDF page and get the text using images_to_text. Args: page: The page object from which to extract images. Returns: str: The extracted text from the images on the page. rrImagez/XObjectz /ResourcesNz/Subtypez/Imagez/Filterrz/Heightz/WidthZdtyper\Unknown PDF Filter!ZPNG)r3z image/pngZ mime_type image_text)$rrPILrrrrAZ get_object_PDF_FILTER_WITHOUT_LOSSnp frombufferget_datauint8reshape_PDF_FILTER_WITH_LOSSarrayopenioBytesIOloggerwarningZ fromarraysaver from_datagetvaluenextrrrr;rs_FORMAT_IMAGE_STRr3 _JOIN_IMAGESr're) rr>rZxObjectrobjnp_imageheightwidth image_bytesr1rrrr"rs@      z$PyPDFParser.extract_images_from_pageNF)rvrwrxrZrpryrqrrrrzrsr{rtr|rur}r1rrr)r>rrr) __name__ __module__ __qualname____doc___DEFAULT_PAGES_DELIMITERrrr __classcell__rrrr"rns2 3MrncspeZdZdZdZ d-ddeddddd.fddZed/ddZed0ddZ !d1d2d&d'Z d3d+d,Z Z S)4PDFMinerParseraParse a blob from a PDF using `pdfminer.six` library. This class provides methods to parse a blob from a PDF document, supporting various configurations such as handling password-protected PDFs, extracting images, and defining extraction mode. It integrates the 'pdfminer.six' library for PDF processing and offers synchronous blob parsing. Examples: Setup: .. code-block:: bash pip install -U langchain-community pdfminer.six pillow Load a blob from a PDF file: .. code-block:: python from langchain_core.documents.base import Blob blob = Blob.from_path("./example_data/layout-parser-paper.pdf") Instantiate the parser: .. code-block:: python from langchain_community.document_loaders.parsers import PDFMinerParser parser = PDFMinerParser( # password = None, mode = "single", pages_delimiter = " ", # extract_images = True, # images_to_text = convert_images_to_text_with_tesseract(), ) Lazily parse the blob: .. code-block:: python docs = [] docs_lazy = parser.lazy_parse(blob) for doc in docs_lazy: docs.append(doc) print(docs[0].page_content[:100]) print(docs[0].metadata) FNrr!)rvrprqrrrsconcatenate_pagesrxrZrvr[rpryrqrrrrzrsr{rOptional[bool]cst|dvr td|r|st}||_||_||_||_||_||_ |dur>t j s5dt _ t d|r9dnd|_dSdS)aHInitialize a parser based on PDFMiner. Args: password: Optional password for opening encrypted PDFs. mode: Extraction mode to use. Either "single" or "page" for page-wise extraction. pages_delimiter: A string delimiter to separate pages in single-mode extraction. extract_images: Whether to extract images from PDF. images_inner_format: The format for the parsed output. - "text" = return the content as is - "markdown-img" = wrap the content into an image markdown link, w/ link pointing to (`![body)(#)`] - "html-img" = wrap the content as the `alt` text of an tag and link to (`{body}`) concatenate_pages: Deprecated. If True, concatenate all PDF pages into one a single document. Otherwise, return one document per page. Returns: This method does not directly return data. Use the `parse` or `lazy_parse` methods to retrieve parsed documents with content and metadata. Raises: ValueError: If the `mode` is not "single" or "page". Warnings: `concatenate_pages` parameter is deprecated. Use `mode='single' or 'page' instead. r~rNTzS`concatenate_pages` parameter is deprecated. Use `mode='single' or 'page'` instead.rr>)rrrBrrxrrrsrvrprqr_warn_concatenate_pagesrr)rrxrvrprqrrrsrrrr"rs& (zPDFMinerParser.__init__sUnion[bytes, str]rcsxddlmt|tr|drt|ddddSzdd |D}d fd d |DWSty;t|YSw) z Decodes a PDFDocEncoding string to Unicode. Adds py3 compatibility to pdfminer's version. Args: s: The string to decode. Returns: str: The decoded Unicode string. rPDFDocEncodingsNzutf-16beignorecss&|]}t|tr t|n|VqdSr^)rCrord)r crrr" Ss$z-PDFMinerParser.decode_text..rc3s|]}|VqdSr^r)r orrr"rTs)Zpdfminer.utilsrrCbytesrMrr' IndexError)rZordsrrr" decode_textBs   zPDFMinerParser.decode_textrrcCsddlm}t|dr|}t|trtttj|St||r't |j St|t t fr3t |St|t rJ|D] \}}t|||<q<|S|S)z Recursively resolve the metadata values. Args: obj: The object to resolve and decode. It can be of any type. Returns: The resolved and decoded object. r) PSLiteralresolve)Zpdfminer.psparserrhasattrrrClistmaprresolve_and_decodernamerrrrK)rrrRrSrrr"rXs      z!PDFMinerParser.resolve_and_decoderTfprcachingr=c Csddlm}m}m}||}||||d}i} |jD]} | | q| D](\} } z t| | | <Wq&t yN} zt d| t | WYd} ~ q&d} ~ wwt t||| d<| S)ag Extract metadata from a PDF file. Args: fp: The file pointer to the PDF file. password: The password for the PDF file, if encrypted. Defaults to an empty string. caching: Whether to cache the PDF structure. Defaults to True. Returns: Metadata of the PDF file. r) PDFDocumentPDFPage PDFParser)rvrzD[WARNING] Metadata key "%s" could not be parsed due to exception: %sNr-)pdfminer.pdfpagerrrinfoupdaterKrr ExceptionrrrrrZ create_pages)rrrvrrrrparserdocr<rrRrSerrr" _get_metadatats&   zPDFMinerParser._get_metadatar1rrc #s,z7ddl}ddlm}ddlm}mmm}m}m m ddl m }m }ddlm} t|jdkr7tdWn tyCtd w|} t| j| jpTd d } |} tj| jpbd d } |j| d <Gfd dd|}t|| || |d}g}t| D]H\}}dd|| }|!}j"dkrLjddt#|t$| d|iBdVq|%dr|dd}|&|qj"dkrj'(|}t#|t$| dVWdn1swYWddSWddS1swYdS)a Lazily parse the blob. Insert image, if possible, between two paragraphs. In this way, a paragraph can be continued on the next page. Args: blob: The blob to parse. Raises: ImportError: If the `pdfminer.six` or `pillow` package is not found. Yield: An iterator over the parsed documents. rN)PDFLayoutAnalyzer)LAParams LTContainerLTImageLTItemLTPageLTText LTTextBox)PDFPageInterpreterPDFResourceManager)ri:>4zThis parser is tested with pdfminer.six version 20201018 or later. Remove pdfminer, and install pdfminer.six with `pip uninstall pdfminer && pip install pdfminer.six`.zMpdfminer package not found, please install it with `pip install pdfminer.six`rrr/cs>eZdZ  ddfd d Zdfdd ZZS)z*PDFMinerParser.lazy_parse..VisitorrNrsrcmgrrpagenorElaparamsOptional[LAParams]rNonecstj|||ddS)N)rr)rr)rrrrrrr"rsz3PDFMinerParser.lazy_parse..Visitor.__init__ltpagerc s(dfdd |dS)Nitemrrrcst|r|D]}|qn t|r|t|r'ddSt|rcjraddlm}|}||}tt |}d|j d<t j |j }t||jdSdS dS)Nr%r) ImageWriterr4r/)rCwriteget_textrrZpdfminer.imagerZ export_imager from_pathrr<rrrr;rs)rchildrZ image_writerfilenamer1r)rrrrrenderrtempdirtext_iorr"rs6         zIPDFMinerParser.lazy_parse..Visitor.receive_layout..render)rrrrr)merrrrrrrr)rr"receive_layouts z9PDFMinerParser.lazy_parse..Visitor.receive_layout)rN)rrrrErrrr)rrrr)rrrrr rrr rr"Visitors "r )rr>r r\r))pdfminerZpdfminer.converterrZpdfminer.layoutrrrrrrrZpdfminer.pdfinterprrrrrErr&rrZ get_pagesrvrTrr/rStringIOrtruncateseekZ process_pagerrQrprrFendswithrrqr')rr1rrrrrrrrrrrrr Zvisitor_for_allZ all_contentir>rhZdocument_contentrr r"rsl $    (           RzPDFMinerParser.lazy_parseF)rxrZrvr[rpryrqrrrrzrsr{rr)rrrr)rrrr)rT)rrrvrrrZrr=r) rrrrrrr staticmethodrrrrrrrrr"rs(2<   .rc seZdZdZeZ  d0ddedddddd1fddZd2dd Z d3d4d!d"Z d5d'd(Z d6d*d+Z d7d,d-Z d8d.d/ZZS)9 PyMuPDFParseraParse a blob from a PDF using `PyMuPDF` library. This class provides methods to parse a blob from a PDF document, supporting various configurations such as handling password-protected PDFs, extracting images, and defining extraction mode. It integrates the 'PyMuPDF' library for PDF processing and offers synchronous blob parsing. Examples: Setup: .. code-block:: bash pip install -U langchain-community pymupdf Load a blob from a PDF file: .. code-block:: python from langchain_core.documents.base import Blob blob = Blob.from_path("./example_data/layout-parser-paper.pdf") Instantiate the parser: .. code-block:: python from langchain_community.document_loaders.parsers import PyMuPDFParser parser = PyMuPDFParser( # password = None, mode = "single", pages_delimiter = " ", # images_parser = TesseractBlobParser(), # extract_tables="markdown", # extract_tables_settings=None, # text_kwargs=None, ) Lazily parse the blob: .. code-block:: python docs = [] docs_lazy = parser.lazy_parse(blob) for doc in docs_lazy: docs.append(doc) print(docs[0].page_content[:100]) print(docs[0].metadata) NFr>r!)rvrprqrrrsextract_tablesextract_tables_settings text_kwargsr}rxrZrvr[rpryrqrrrrzrsr{r/Union[Literal['csv', 'markdown', 'html'], None]rrrc szt|dvr td|r|dvrtd||_||_||_|p#i|_|r,|s,t}||_||_ ||_ ||_ | |_ dS)aInitialize a parser based on PyMuPDF. Args: password: Optional password for opening encrypted PDFs. mode: The extraction mode, either "single" for the entire document or "page" for page-wise extraction. pages_delimiter: A string delimiter to separate pages in single-mode extraction. extract_images: Whether to extract images from the PDF. images_parser: Optional image blob parser. images_inner_format: The format for the parsed output. - "text" = return the content as is - "markdown-img" = wrap the content into an image markdown link, w/ link pointing to (`![body)(#)`] - "html-img" = wrap the content as the `alt` text of an tag and link to (`{body}`) extract_tables: Whether to extract tables in a specific format, such as "csv", "markdown", or "html". extract_tables_settings: Optional dictionary of settings for customizing table extraction. Returns: This method does not directly return data. Use the `parse` or `lazy_parse` methods to retrieve parsed documents with content and metadata. Raises: ValueError: If the mode is not "single" or "page". ValueError: If the extract_tables format is not "markdown", "html", or "csv". r~r)markdownr9csvzmode must be markdownN) rrrBrprqrvrrrxrsrrrr) rrrxrvrprqrrrsrrrrr"rVs +   zPyMuPDFParser.__init__r1rrcCs ||Sr^) _lazy_parse)rr1rrr"rszPyMuPDFParser.lazy_parsec cszXddl}|p |j}|jsXddlm}m}m}m}iddddddddd dd |d dd dd |ddddddd|d|dddddddddddd|_Wn tydtdwt j | d}|j dury| |} n|j |dd} | jr| |j|| |} g} | D]$} || | |} |jdkrt| t| d| jiBdVq| | q|jdkrt|j| t| dVWdn1swYWddSWddS1swYdS)a Lazily parse the blob. Insert image, if possible, between two paragraphs. In this way, a paragraph can be continued on the next page. Args: blob: The blob to parse. text_kwargs: Optional keyword arguments to pass to the `get_text` method. If provided at run time, it will override the default text_kwargs. Raises: ImportError: If the `pypdf` package is not found. Yield: An iterator over the parsed documents. rN)DEFAULT_JOIN_TOLERANCEDEFAULT_MIN_WORDS_HORIZONTALDEFAULT_MIN_WORDS_VERTICALDEFAULT_SNAP_TOLERANCEZclipZvertical_strategylinesZhorizontal_strategyZvertical_linesZhorizontal_linesZsnap_toleranceZsnap_x_toleranceZsnap_y_toleranceZjoin_toleranceZjoin_x_toleranceZjoin_y_toleranceZedge_min_lengthZmin_words_verticalZmin_words_horizontalZintersection_toleranceZintersection_x_toleranceZintersection_y_tolerance)Ztext_toleranceZtext_x_toleranceZtext_y_toleranceZstrategyZ add_lineszGpymupdf package not found, please install it with `pip install pymupdf`Zpdf)streamZfiletyper>rr)pymupdfrrZ pymupdf.tablerrr r!r&r_lockrdatar is_encryptedZ authenticaterv_extract_metadata_get_page_contentrQrprrFnumberrrqr')rr1rr%rrr r!rGrr full_contentr>rhrrr"rs                  "zPyMuPDFParser._lazy_parserpymupdf.Document pymupdf.Pager=c Cs^|jdii|j|}|||}||}g}|r!|||r(||t||}|S)a:Get the text of the page using PyMuPDF and RapidOCR and issue a warning if it is empty. Args: doc: The PyMuPDF document object. page: The PyMuPDF page object. blob: The blob being parsed. Returns: str: The text content of the page. Nr)rr_extract_images_from_page_extract_tables_from_pagerrm) rrr>rrXrZtables_from_pagerVrhrrr"r*s     zPyMuPDFParser._get_page_contentrc s\tiddd|j|jtdfddjD}dD]}|jvr+j|||<q|S)zExtract metadata from the document and page. Args: doc: The PyMuPDF document object. blob: The blob being parsed. Returns: dict: The extracted metadata. ZPyMuPDFr)r,r0r.r/rGr-cs,i|]}tj|ttfr|j|qSr)rCr<rrEr rRrrr" &s  z3PyMuPDFParser._extract_metadata..)ZmodDateZ creationDate)rTr/rr<)rrr1r<rRrr2r"r)s(   zPyMuPDFParser._extract_metadatac Cs|jsdSddl}|}g}|D]E}|jrV|d}|||}tj|jtjd|j |j d} t } t | | tj| dd} t|j| j} |t| | |jqtjttd|dS) a Extract images from a PDF page and get the text using images_to_text. Args: doc: The PyMuPDF document object. page: The PyMuPDF page object. Returns: str: The extracted text from the images on the page. rrNrr\application/x-npyrr)rrr%Z get_imagesZPixmaprrZsamplesrrrrrrnumpyrrrrrrrrr;rsrr3rr're) rrr>r%Zimg_listrr(ZxrefZpiximagerr1rrrr"r/2s2     z'PyMuPDFParser._extract_images_from_pagecCs|jdurdSddl}t|jj|fi|j}|rP|jdkr)tdd|DS|jdkr8tdd|DS|jd krGtd d|DStd |jd dS) zExtract tables from a PDF page. Args: page: The PyMuPDF page object. Returns: str: The extracted tables in the specified format. NrrrcSsg|]}|qSr)Z to_markdownr tablerrr"r#kr$z;PyMuPDFParser._extract_tables_from_page..r9cSs g|] }|jddddqS)F)headerindexZ bold_rows) to_pandasZto_htmlr7rrr"r#nsrcSsg|] }|jdddqS)F)r9r:)r;Zto_csvr7rrr"r#yszextract_tables z not implemented) rr%rr8Z find_tablesr _JOIN_TABLESr'rB)rr>r%Z tables_listrrr"r0Ys2     z'PyMuPDFParser._extract_tables_from_pager)rr}rxrZrvr[rpryrqrrrrzrsr{rrrr}rrrr^)r1rrr}rr)rr-r>r.rr=rr)rr-r1rrr)rr-r>r.rr)r>r.rr)rrrr threadingLockr&rrrrr*r)r/r0rrrrr"rs*6 = ]  'rcsPeZdZdZeZ dddedddd fddZd!ddZ d"ddZ Z S)#PyPDFium2ParseraoParse a blob from a PDF using `PyPDFium2` library. This class provides methods to parse a blob from a PDF document, supporting various configurations such as handling password-protected PDFs, extracting images, and defining extraction mode. It integrates the 'PyPDFium2' library for PDF processing and offers synchronous blob parsing. Examples: Setup: .. code-block:: bash pip install -U langchain-community pypdfium2 Load a blob from a PDF file: .. code-block:: python from langchain_core.documents.base import Blob blob = Blob.from_path("./example_data/layout-parser-paper.pdf") Instantiate the parser: .. code-block:: python from langchain_community.document_loaders.parsers import PyPDFium2Parser parser = PyPDFium2Parser( # password=None, mode="page", pages_delimiter=" ", # extract_images = True, # images_to_text = convert_images_to_text_with_tesseract(), ) Lazily parse the blob: .. code-block:: python docs = [] docs_lazy = parser.lazy_parse(blob) for doc in docs_lazy: docs.append(doc) print(docs[0].page_content[:100]) print(docs[0].metadata) FNr>r!)rvrprqrrrsrxrZrvr[rpryrqrrrrzrsr{rrcsPt|dvr td||_|r|st}||_||_||_||_||_ dS)ukInitialize a parser based on PyPDFium2. Args: password: Optional password for opening encrypted PDFs. mode: The extraction mode, either "single" for the entire document or "page" for page-wise extraction. pages_delimiter: A string delimiter to separate pages in single-mode extraction. extract_images: Whether to extract images from the PDF. images_parser: Optional image blob parser. images_inner_format: The format for the parsed output. - "text" = return the content as is - "markdown-img" = wrap the content into an image markdown link, w/ link pointing to (`![body)(#)`] - "html-img" = wrap the content as the `alt` text of an tag and link to (`{body}`) extraction_mode: “plain” for legacy functionality, “layout” for experimental layout mode functionality extraction_kwargs: Optional additional parameters for the extraction process. Returns: This method does not directly return data. Use the `parse` or `lazy_parse` methods to retrieve parsed documents with content and metadata. Raises: ValueError: If the mode is not "single" or "page". r~rN) rrrBrxrrrrsrvrprq)rrxrvrprqrrrsrrr"rs & zPyPDFium2Parser.__init__r1rrc cszddl}Wn tytdwtj|}d}z|j||jdd}g}t|}|j |d<t ||d<t |D]I\}}| } d | } | ||} t| g| } ||jd kr| dss| d7} t| ti|d |id Vq?|| q?|jd krt|j |t|d VW|r|n|r|wwWdn1swYWddSWddS1swYdS) rrNzKpypdfium2 package not found, please install it with `pip install pypdfium2`T)rvZ autocloser/r-r%r>rr) pypdfium2r&r?r&rZ PdfDocumentrvrTZget_metadata_dictr/rrZ get_textpager'Zget_text_range splitlinescloser/rmrQrprrrFrrq) rr1r@rGrr,rrr>Z text_pagerXZimage_from_pagerhrrr"rs~             "zPyPDFium2Parser.lazy_parsepypdfium2._helpers.page.PdfPagec Cs|jsdSddlm}t|j|jfd}|sdSg}|D]<}t}| }|j dkr/qt || t j|dd}t|j|j} |t|| |j|qtjt|dS) rrrN)rer#r4rr)rrZ pypdfium2.rawrawrZ get_objectsZFPDF_PAGEOBJ_IMAGErrZ get_bitmapZto_numpysizer5rrrrrrrrr;rsrBrr3rr') rr>Zpdfium_crZ str_imagesr6rrr1Ztext_from_imagerrr"r/<s(      z)PyPDFium2Parser._extract_images_from_pager)rxrZrvr[rpryrqrrrrzrsr{rrr)r>rCrr) rrrrr=r>r&rrrr/rrrrr"r?s4 2Kr?c@s@eZdZdZ   ddd d ZdddZdddZdddZdS)PDFPlumberParserzParse `PDF` with `PDFPlumber`.NFrOptional[Mapping[str, Any]]deduperZrxrrcCs>zddl}Wn tytdw|pi|_||_||_dS)zInitialize the parser. Args: text_kwargs: Keyword arguments to pass to ``pdfplumber.Page.extract_text()`` dedupe: Avoiding the error of duplicate characters if `dedupe=True`. rNzEpillow package not found, please install it with `pip install pillow`)rr&rrHrx)rrrHrxrrrr"r`s    zPDFPlumberParser.__init__r1rrc#s`ddl}}||fddjDEdHWddS1s)wYdS)Lazily parse the blob.rNc sbg|]-}t|d|tjj|jdtjdfifddjDdqS)r%r)r/rGr>r-cs.i|]}tj|ttfvr|j|qSr)rLr<rrEr1r2rr"r3s  z:PDFPlumberParser.lazy_parse...r) r_process_page_contentr/rr/rrrr<)r r>r1rrrr"r#}s* z/PDFPlumberParser.lazy_parse..) pdfplumberrrr)rr1rLrGrrKr"rvs  "zPDFPlumberParser.lazy_parser>pdfplumber.page.PagercCs.|jr|jdi|jS|jdi|jS)z)Process the page content based on dedupe.Nr)rHZ dedupe_charsrr)rr>rrr"rJsz&PDFPlumberParser._process_page_contentc Csddlm}|js dSg}|jD]g}|ddjtvr_|dddkrB|t| d|dd |dd f|d  d q|tj |d tj d |dd |dd d q|ddjtvrr||d qtdqt|S)z8Extract images from page and get the text with RapidOCR.rrrr$FilterZBitsPerComponentr1ZWidthZHeightLrr\r)rrrxrrrrrr frombytesrconvertrrrrwarningswarnr+)rr>rrr(rrr"r/s4     z*PDFPlumberParser._extract_images_from_page)NFF)rrGrHrZrxrZrrr)r>rMrr)rrrrrrrJr/rrrr"rF]s  rFc@s0eZdZdZ  ddddd d ZdddZdS)AmazonTextractPDFParsera{Send `PDF` files to `Amazon Textract` and parse them. For parsing multi-page PDFs, they have to reside on S3. The AmazonTextractPDFLoader calls the [Amazon Textract Service](https://aws.amazon.com/textract/) to convert PDFs into a Document structure. Single and multi-page documents are supported with up to 3000 pages and 512 MB of size. For the call to be successful an AWS account is required, similar to the [AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-configure.html) requirements. Besides the AWS configuration, it is very similar to the other PDF loaders, while also supporting JPEG, PNG and TIFF and non-native PDF formats. ```python from langchain_community.document_loaders import AmazonTextractPDFLoader loader=AmazonTextractPDFLoader("example_data/alejandro_rosalez_sample-small.jpeg") documents = loader.load() ``` One feature is the linearization of the output. When using the features LAYOUT, FORMS or TABLES together with Textract ```python from langchain_community.document_loaders import AmazonTextractPDFLoader # you can mix and match each of the features loader=AmazonTextractPDFLoader( "example_data/alejandro_rosalez_sample-small.jpeg", textract_features=["TABLES", "LAYOUT"]) documents = loader.load() ``` it will generate output that formats the text in reading order and try to output the information in a tabular structure or output the key/value pairs with a colon (key: value). This helps most LLMs to achieve better accuracy when processing these texts. N)linearization_configtextract_featuresOptional[Sequence[int]]client Optional[Any]rV!Optional[TextLinearizationConfig]rrcsz:ddlddlmm}|_||_|dur#fdd|D|_ng|_|dur.||_n |jjddddd |_Wn t yEt d w|saz ddl }| d |_ WdSt y`t d w||_ dS) a5Initializes the parser. Args: textract_features: Features to be used for extraction, each feature should be passed as an int that conforms to the enum `Textract_Features`, see `amazon-textract-caller` pkg client: boto3 textract client linearization_config: Config to be used for linearization of the output should be an instance of TextLinearizationConfig from the `textractor` pkg rNcsg|]}|qSr)ZTextract_Features)r ftcrr"r#s z4AmazonTextractPDFParser.__init__..Tz# z## *)Zhide_figure_layoutZ title_prefixZsection_header_prefixZlist_element_prefixzCould not import amazon-textract-caller or amazon-textract-textractor python package. Please install it with `pip install amazon-textract-caller` & `pip install amazon-textract-textractor`.ZtextractzRCould not import boto3 python package. Please install it with `pip install boto3`.) ZtextractcallerZtextractor.entities.documententitiesdocumentr^ textractorrWrVrr&boto3rYboto3_textract_client)rrWrYrVrbrcrr]r"rsD     z AmazonTextractPDFParser.__init__r1rrccs|jr tt|jnd}|r&|jdkr&|jr&|jjt|j|j|jd}n|jj| |j|jj j |jd}|j j |}t|jD]\}}t |j|jd|j|dddVqCdS) zIterates over the Blob pages and returns an Iterator with a Document for each page, like the other parsers If multi-page document, blob.path has to be set to the S3 URI and for single page docs the blob.data is taken NZs3)input_documentfeaturesrd)rerfZ call_moderd)configrr/r>r)pathrrschemenetlocr^Z call_textractrWrdas_bytesZTextract_Call_ModeZ FORCE_SYNCrbrrrrrrVr/)rr1Zurl_parse_resultZtextract_response_jsonraidxr>rrr"r(s2   z"AmazonTextractPDFParser.lazy_parse)NN)rWrXrYrZrVr[rrr)rrrrrrrrrr"rUs/?rUc@s.eZdZdZdddZdd dZdddZdS)DocumentIntelligenceParserzjLoads a PDF with Azure Document Intelligence (formerly Form Recognizer) and chunks at character level.rYrmodelrcCstd||_||_dS)Na<langchain_community.document_loaders.parsers.pdf.DocumentIntelligenceParserand langchain_community.document_loaders.pdf.DocumentIntelligenceLoader are deprecated. Please upgrade to langchain_community.document_loaders.DocumentIntelligenceLoader for any file parsing purpose using Azure Document Intelligence service.)rSrTrYro)rrYrorrr"rPs  z#DocumentIntelligenceParser.__init__r1rr)rrccsD|jD]}ddd|jD}t||j|jdd}|VqdS)N cSsg|]}|jqSr)r2)r linerrr"r#^sz=DocumentIntelligenceParser._generate_docs..rhr)rr'r"rr/r)rr1r)pr2drrr"_generate_docs\s z)DocumentIntelligenceParser._generate_docsccs^| }|j|j|}|}|||}|EdHWddS1s(wYdS)rIN)rrYZbegin_analyze_documentror)rt)rr1Zfile_objZpollerr)docsrrr"ris   "z%DocumentIntelligenceParser.lazy_parseN)rYrror)r1rr)rrrr)rrrrrrtrrrrr"rnLs    rn)rrrr)r1rr2rr3rrr)r<r=rr=)rVrWrXrrr)Cr __future__rr9rloggingr=rSrpathlibrtempfilertypingrrrr r r r r rrr urllib.parserr5rZlangchain_core.documentsrZ)langchain_community.document_loaders.baserZ1langchain_community.document_loaders.blob_loadersrZ3langchain_community.document_loaders.parsers.imagesrrrLr%rr@Z)textractor.data.text_linearization_configrrrr+ getLoggerrrrrr<rr?r;rFrTrcrmrnrrr?rFrUrnrrrr"sl    4          ' 5_NmV^