o Zhm@sddlZddlmZddlmZddlmZmZmZm Z m Z m Z m Z ddl mZddlmZddlmZddlmZmZGd d d eZGd d d eZdS) N) TextIOWrapper)Path)AnyDictIteratorListOptionalSequenceUnion)Document) BaseLoader)detect_file_encodings)UnstructuredFileLoadervalidate_unstructured_versionc@seZdZdZ     ddddeeefdeedeed ee d eed e d eefd dZ de e fddZdede e fddZdS) CSVLoaderaLoad a `CSV` file into a list of Documents. Each document represents one row of the CSV file. Every row is converted into a key/value pair and outputted to a new line in the document's page_content. The source for each document loaded from csv is set to the value of the `file_path` argument for all documents by default. You can override this by setting the `source_column` argument to the name of a column in the CSV file. The source of each document will then be set to the value of the column with the name specified in `source_column`. Output Example: .. code-block:: txt column1: value1 column2: value2 column3: value3 Instantiate: .. code-block:: python from langchain_community.document_loaders import CSVLoader loader = CSVLoader(file_path='./hw_200.csv', csv_args={ 'delimiter': ',', 'quotechar': '"', 'fieldnames': ['Index', 'Height', 'Weight'] }) Load: .. code-block:: python docs = loader.load() print(docs[0].page_content[:100]) print(docs[0].metadata) .. code-block:: python Index: Index Height: Height(Inches)" Weight: "Weight(Pounds)" {'source': './hw_200.csv', 'row': 0} Async load: .. code-block:: python docs = await loader.aload() print(docs[0].page_content[:100]) print(docs[0].metadata) .. code-block:: python Index: Index Height: Height(Inches)" Weight: "Weight(Pounds)" {'source': './hw_200.csv', 'row': 0} Lazy load: .. code-block:: python docs = [] docs_lazy = loader.lazy_load() # async variant: # docs_lazy = await loader.alazy_load() for doc in docs_lazy: docs.append(doc) print(docs[0].page_content[:100]) print(docs[0].metadata) .. code-block:: python Index: Index Height: Height(Inches)" Weight: "Weight(Pounds)" {'source': './hw_200.csv', 'row': 0} NF)content_columns file_path source_columnmetadata_columnscsv_argsencodingautodetect_encodingrcCs2||_||_||_||_|pi|_||_||_dS)a Args: file_path: The path to the CSV file. source_column: The name of the column in the CSV file to use as the source. Optional. Defaults to None. metadata_columns: A sequence of column names to use as metadata. Optional. csv_args: A dictionary of arguments to pass to the csv.DictReader. Optional. Defaults to None. encoding: The encoding of the CSV file. Optional. Defaults to None. autodetect_encoding: Whether to try to autodetect the file encoding. content_columns: A sequence of column names to use for the document content. If not present, use all columns that are not part of the metadata. N)rrrrrrr)selfrrrrrrrrrf/var/www/html/lang_env/lib/python3.10/site-packages/langchain_community/document_loaders/csv_loader.py__init__cs  zCSVLoader.__init__returnc cs8z&t|jd|jd}||EdHWdWdS1s wYWdSty}zT|jrlt|j}|D]1}z&t|jd|jd}||EdH WdWn$1s[wYWq9tyjYq9wn td|j|WYd}~dSWYd}~dSd}~wty}z td|j|d}~ww)N)newlinerzError loading ) openrr_CSVLoader__read_fileUnicodeDecodeErrorrr RuntimeError Exception)rcsvfileeZdetected_encodingsrrrr lazy_loads:&  &   zCSVLoader.lazy_loadr$c #stj|fij}t|D]]\}}zjdur|jntj}Wnty4tdjdwd fdd| D}||d}j D]}z||||<WqKtydtd|dwt ||dVqdS) NzSource column 'z' not found in CSV file. c3s||]9\}}jr|jvr;n|jvr|dur|n|dt|tr(|nt|tr6dttj|n|VqdS)Nz: ,)rrstrip isinstancestrlistjoinmap).0kvrrr s    z(CSVLoader.__read_file..)sourcerowzMetadata column ')Z page_contentmetadata) csv DictReaderr enumeraterr+rKeyError ValueErrorr-itemsrr ) rr$Z csv_readerir5r4contentr6colrr2rZ __read_files2       zCSVLoader.__read_file)NrNNF)__name__ __module__ __qualname____doc__r r+rrr rboolrrr r&rr rrrrrs4U   !rcs@eZdZdZ d dededeffdd Zdefd d ZZ S) UnstructuredCSVLoadera|Load `CSV` files using `Unstructured`. Like other Unstructured loaders, UnstructuredCSVLoader can be used in both "single" and "elements" mode. If you use the loader in "elements" mode, the CSV file will be a single Unstructured Table element. If you use the loader in "elements" mode, an HTML representation of the table will be available in the "text_as_html" key in the document metadata. Examples -------- from langchain_community.document_loaders.csv_loader import UnstructuredCSVLoader loader = UnstructuredCSVLoader("stanley-cups.csv", mode="elements") docs = loader.load() singlermodeunstructured_kwargsc s&tddtjd||d|dS)a Args: file_path: The path to the CSV file. mode: The mode to use when loading the CSV file. Optional. Defaults to "single". **unstructured_kwargs: Keyword arguments to pass to unstructured. z0.6.8)Zmin_unstructured_version)rrGNr)rsuperr)rrrGrH __class__rrrs zUnstructuredCSVLoader.__init__rcCs"ddlm}|dd|ji|jS)Nr) partition_csvfilenamer)Zunstructured.partition.csvrLrrH)rrLrrr _get_elementss z#UnstructuredCSVLoader._get_elements)rF) r@rArBrCr+rrrrN __classcell__rrrJrrEsrE)r7iorpathlibrtypingrrrrrr r Zlangchain_core.documentsr Z)langchain_community.document_loaders.baser Z,langchain_community.document_loaders.helpersr Z1langchain_community.document_loaders.unstructuredrrrrErrrrs  $   /