o Zh%J@sdZddlZddlZddlZddlmZddlmZmZm Z m Z m Z m Z ddl mZddlmZddlmZddlmZdd lmZerVdd lmZdd lmZdd lmZeeZeGd ddZ GdddeZ!dS)zModule contains a PDF parser based on Document AI from Google Cloud. You need to install two libraries to use this parser: pip install google-cloud-documentai pip install google-cloud-documentai-toolbox N) dataclass) TYPE_CHECKINGAnyIteratorListOptionalSequence)BaseBlobParser)Blob)Document) batch_iterate)get_client_info) OperationDocumentProcessorServiceClient)ProcessOptionsc@s"eZdZUdZeed<eed<dS)DocAIParsingResultsz1A dataclass to store Document AI parsing results. source_path parsed_pathN)__name__ __module__ __qualname____doc__str__annotations__rrW/var/www/html/lang_env/lib/python3.10/site-packages/langchain_google_community/docai.pyrs  rc@seZdZdZdddddddeddeedeedeed eef d d Zd ed ee fddZ    d5dee dee e dee dee d df ddZ d6d edeeded ee fddZ   d7deedeed e d!e ded ee f d"d#Zd$e ed ee fd%d&Zd'e ed e d(fd)d*Zd+e d(d e fd,d-Zddd.dd/deedeed eed0e deeded e d(fd1d2Zd+e d(d e efd3d4ZdS)8 DocAIParserz`Google Cloud Document AI` parser. For a detailed explanation of Document AI, refer to the product documentation. https://cloud.google.com/document-ai/docs/overview N)client project_idlocationgcs_output_pathprocessor_namerrrr r!r"c Cst|t|kr tdd}|rt||std|d||_||_|r+||_d Szddlm}ddl m }Wnt yJ} zt d| d } ~ ww|||d d } || t d d d|_|jj |dj|_|jdkrqd|_d Sd|_d S)aInitializes the parser. Args: client: a DocumentProcessorServiceClient to use location: a Google Cloud location where a Document AI processor is located gcs_output_path: a path on Google Cloud Storage to store parsing results processor_name: full resource name of a Document AI processor or processor version You should provide either a client or location (and then a client would be instantiated). zGYou must specify either a client or a location to instantiate a client.z?projects\/[0-9]+\/locations\/[a-z\-0-9]+\/processors\/[a-z0-9]+zProcessor name z has the wrong format. If your prediction endpoint looks like https://us-documentai.googleapis.com/v1/projects/PROJECT_ID/locations/us/processors/PROCESSOR_ID:process, use only projects/PROJECT_ID/locations/us/processors/PROCESSOR_ID part.r) ClientOptionsrCould not import google-cloud-documentai python package. Please, install docai dependency group: `pip install langchain-google-community[docai]`Nz-documentai.googleapis.com)Zquota_project_idZ api_endpointz document-ai)module)Zclient_optionsZ client_infonameZLAYOUT_PARSER_PROCESSORTF)bool ValueErrorre fullmatch_gcs_output_path_processor_name_clientZgoogle.api_core.client_optionsr#google.cloud.documentair ImportErrorr Z get_processortypeZ_processor_type_use_layout_parser) selfrrr r!r"patternr#rexcoptionsrrr__init__.sH      zDocAIParser.__init__blobreturnccs|j|g|jdEdHdS)zParses a blob lazily. Args: blobs: a Blob to parse This is a long-running operation. A recommended way is to batch documents together and use the `batch_parse()` method. )r!N) batch_parser,)r3r8rrr lazy_parseqs zDocAIParser.lazy_parseTenable_native_pdf_parsing page_range chunk_sizeinclude_ancestor_headingsrc Csz ddlm}m}Wnty}ztd|d}~ww|jr=|j|jj||dd}|r3|j|dnd} ||| d} | S|rD||d nd} |rN|j|dnd} || | d } | S) aMPrepare process options for DocAI process request Args: enable_native_pdf_parsing: enable pdf embedded text extraction page_range: list of page numbers to parse. If `None`, entire document will be parsed. chunk_size: maximum number of characters per chunk (supported only with Document AI Layout Parser processor). include_ancestor_headings: whether or not to include ancestor headings when splitting (supported only with Document AI Layout Parser processor). r) OcrConfigrddocumentai package not found, please install it with `pip install langchain-google-community[docai]`N)r?r@)Zchunking_config)pages) layout_configindividual_page_selector)r=) ocr_configrE) google.cloud.documentai_v1.typesrArr0r2Z LayoutConfigZChunkingConfigZIndividualPageSelector) r3r=r>r?r@rArr5rDrEprocess_optionsrFrrr_prepare_process_options|sL z$DocAIParser._prepare_process_options field_maskprocess_options_kwargsc +szddlm}Wnty}ztd|d}~wwzddlmWnty4}ztd|d}~ww|jdi|}|j|j|j |j j j pLdd|d |d |j rifd d jjjDEdHdSfd d jjDEdHdS)aParses a blob lazily using online processing. Args: blob: a blob to parse. field_mask: a comma-separated list of which fields to include in the Document AI response. suggested: "text,pages.pageNumber,pages.layout" process_options_kwargs: optional parameters to pass to the Document AI processors r documentair$N_text_from_layoutldocumentai_toolbox package not found, please install it with `pip install langchain-google-community[docai]`application/pdfgcs_uriZ mime_typeT)r'Z gcs_documentrHskip_human_reviewrJc3s(|]}t|j|jjddVqdS)chunk_idsourceZ page_contentmetadataN)r contentrVpath).0chunk)r8rr s z-DocAIParser.online_process..c3s2|]}t|jjj|jjddVqdS)pagerWrXN)r layoutdocumenttext page_numberr[)r\r`rOr8responserrr^s r) google.cloudrMr0-google.cloud.documentai_toolbox.wrappers.pagerOrIr.Zprocess_documentZProcessRequestr- GcsDocumentr[mimetyper2rbchunked_documentchunksrC)r3r8rJrKrMr5rHrreronline_processsR  zDocAIParser.online_process<blobs timeout_seccheck_in_interval_secc ks|p|j}|s td|j|fd|i|}dd|D}td|d} ||rJt|| |7} | |kr@td|dtd ||s+|j |d } | | Ed Hd S) aParses a list of blobs lazily. Args: blobs: a list of blobs to parse. gcs_output_path: a path on Google Cloud Storage to store parsing results. timeout_sec: a timeout to wait for Document AI to complete, in seconds. check_in_interval_sec: an interval to wait until next check whether parsing operations have been completed, in seconds. process_options_kwargs: optional parameters to pass to the Document AI processors This is a long-running operation. A recommended way is to decouple parsing from creating LangChain Documents: >>> operations = parser.docai_parse(blobs, gcs_path) >>> parser.is_running(operations) You can get operations names and save them: >>> names = [op.operation.name for op in operations] And when all operations are finished, you can use their results: >>> operations = parser.operations_from_names(operation_names) >>> results = parser.get_results(operations) >>> docs = parser.parse_from_results(results) :An output path on Google Cloud Storage should be provided.r!cSsg|]}|jjqSr)Z operationr'r\oprrr &sz+DocAIParser.batch_parse..z9Started parsing with Document AI, submitted operations %srz#Timeout exceeded! Check operations z later!.) operationsN) r,r) docai_parseloggerdebug is_runningtimesleep TimeoutError get_resultsparse_from_results) r3rpr!rqrrrK output_pathrxoperation_namesZ time_elapsedresultsrrrr:s:      zDocAIParser.batch_parserc #szddlm}ddlm}ddlmWnty&}ztd|d}~ww|D]-|j\}}|||d}|jrIfdd|DEdHq)fd d|DEdHq)dS) Nr) split_gcs_uri) _get_shardsrNrP/c3s6|]}|jjD]}t|j|jjddVqqdSrU)rkrlr rZrVr)r\shardr])resultrrr^Ks  z1DocAIParser.parse_from_results..c3s<|]}|jD]}t|j|j|jjddVqqdSr_)rCr rarcrdr)r\rr`rOrrrr^Ws   ) Z7google.cloud.documentai_toolbox.utilities.gcs_utilitiesrZ1google.cloud.documentai_toolbox.wrappers.documentrrhrOr0rr2)r3rrrr5Zgcs_bucket_nameZ gcs_prefixZshardsrrrr7s0    zDocAIParser.parse_from_resultsrrc sHzddlmWnty}ztd|d}~wwfdd|DS)z5Initializes Long-Running Operations from their names.r)GetOperationRequestzplong running operations package not found, please install it with`pip install langchain-google-community[docai]`Ncs g|] }jj|ddqS)r&)request)r.Z get_operation)r\r'rr3rrrvosz5DocAIParser.operations_from_names..)Z!google.longrunning.operations_pb2rr0)r3rr5rrroperations_from_namescs z!DocAIParser.operations_from_namesrxcCstdd|DS)Ncss|]}| VqdSN)donertrrrr^usz)DocAIParser.is_running..)any)r3rxrrrr|tszDocAIParser.is_runningi)r!r" batch_sizerJrc szddlmWnty}ztd|d}~ww|p|j}|dur'td|p+|j}|dur4tdg} t||dD]8} jjfdd | Dd d } j j j ||d d } |j di|} | |j j|| | | ddq<| S)a[Runs Google Document AI PDF Batch Processing on a list of blobs. Args: blobs: a list of blobs to be parsed gcs_output_path: a path (folder) on GCS to store results processor_name: name of a Document AI processor. batch_size: amount of documents per batch field_mask: a comma-separated list of which fields to include in the Document AI response. suggested: "text,pages.pageNumber,pages.layout" process_options_kwargs: optional parameters to pass to the Document AI processors Document AI has a 1000 file limit per batch, so batches larger than that need to be split into multiple requests. Batch processing is an async long-running operation and results are stored in a output GCS bucket. rrLrBNrsz0A Document AI processor name should be provided.)sizeiterablecs"g|] }j|j|jp ddqS)rQrR)rir[rj)r\r8rLrrrvsz+DocAIParser.docai_parse..)Z documents)Z gcs_documents)rSrJ)Zgcs_output_configT)r'Zinput_documentsZdocument_output_configrHrTr)rgrMr0r,r)r-r ZBatchDocumentsInputConfigZ GcsDocumentsZDocumentOutputConfigZGcsOutputConfigrIappendr.Zbatch_process_documentsZBatchProcessRequest)r3rpr!r"rrJrKr5rrxbatchZ input_configZ output_configrHrrLrrywsZ     zDocAIParser.docai_parsec sFzddlmWnty}ztd|d}~wwfdd|DS)NrBatchProcessMetadatarBcsFg|]}t|jr|jjn|jjjD] }t|j|jdqqS))rr) isinstancerYZindividual_process_statusesZ deserializevaluerZinput_gcs_sourceZoutput_gcs_destination)r\rustatusrrrrvs  z+DocAIParser.get_results..)Zgoogle.cloud.documentai_v1rr0)r3rxr5rrrrs zDocAIParser.get_results)TNr<Tr)Nrnro)rrrrrrr7r rr r;r(rintrIrrmrr:rrrr|ryrrrrrr's  C   > L 7 , Pr)"rloggingr*r} dataclassesrtypingrrrrrrZlangchain_core.document_loadersr Z,langchain_core.document_loaders.blob_loadersr Zlangchain_core.documentsr Zlangchain_core.utils.iterr Z!langchain_google_community._utilsr Zgoogle.api_core.operationrr/rrGr getLoggerrrzrrrrrrs&