o Zh\@snddlmZddlZddlmZmZmZmZmZddl m Z m Z Gddde Z dddZ Gddde ZdS)) annotationsN)AnyListLiteralOptionalUnion)Language TextSplittercs0eZdZdZ ddfd d ZdddZZS)CharacterTextSplitterz(Splitting text that looks at characters. F separatorstris_separator_regexboolkwargsrreturnNonec s"tjdi|||_||_dS)Create a new TextSplitter.N)super__init__ _separator_is_separator_regex)selfr rr __class__rY/var/www/html/lang_env/lib/python3.10/site-packages/langchain_text_splitters/character.pyr s zCharacterTextSplitter.__init__text List[str]cCsB|jr|jnt|j}t|||j}|jrdn|j}|||S)&Split incoming text and return chunks.)rrreescape_split_text_with_regex_keep_separator _merge_splits)rrr splitsrrrr split_texts  z CharacterTextSplitter.split_text)r F)r r rrrrrrrr rr)__name__ __module__ __qualname____doc__rr' __classcell__rrrrr s r rr r keep_separator$Union[bool, Literal['start', 'end']]rrcs|r\|rUtd|d||dkr#fddtdtddDnfd dtdtdD}tddkrB|d d7}|dkrM|d gndg|}n t||}nt|}d d|DS) N()endc g|] }||dqSr.0iZ_splitsrr ( z*_split_text_with_regex..rr5cr3r4rr6r9rrr:*r;cSsg|]}|dkr|qS)r r)r7srrrr:7s)r!splitrangelenlist)rr r.r&rr9rr#s " r#csZeZdZdZ   ddfdd Zd ddZd!ddZed"ddZe d#ddZ Z S)$RecursiveCharacterTextSplitterzSplitting text by recursively look at characters. Recursively tries to split by different characters to find one that works. NTF separatorsOptional[List[str]]r.r/rrrrrrc s.tjdd|i||pgd|_||_dS)rr.)r   r Nr)rr _separatorsr)rrDr.rrrrrrAs z'RecursiveCharacterTextSplitter.__init__rr rcCs&g}|d}g}t|D](\}}|jr|nt|}|dkr"|}nt||r4|}||dd}nq |jr:|nt|}t|||j} g} |jrMdn|}| D]2} || |jkra| | qQ| rp| | |} | | g} |sx| | qQ| | |} | | qQ| r| | |} | | |S)rr=r r5N) enumeraterr!r"searchr#r$Z_length_function _chunk_sizeappendr%extend _split_text)rrrDZ final_chunksr Znew_separatorsr8Z_srr&Z _good_splitsr>Z merged_textZ other_inforrrrNMs@         z*RecursiveCharacterTextSplitter._split_textcCs|||jS)zSplit the input text into smaller chunks based on predefined separators. Args: text (str): The input text to be split. Returns: List[str]: A list of text chunks obtained after splitting. )rNrH)rrrrrr'us z)RecursiveCharacterTextSplitter.split_textlanguagercKs||}|d|dd|S)aReturn an instance of this class based on a specific language. This method initializes the text splitter with language-specific separators. Args: language (Language): The language to configure the text splitter for. **kwargs (Any): Additional keyword arguments to customize the splitter. Returns: RecursiveCharacterTextSplitter: An instance of the text splitter configured for the specified language. T)rDrNr)get_separators_for_language)clsrOrrDrrr from_languages z,RecursiveCharacterTextSplitter.from_languagecCs|tjks |tjkrgdS|tjkrgdS|tjkr gdS|tjkr)gdS|tjkr2gdS|tjkr;gdS|tjkrDgdS|tj krMgdS|tj krVgd S|tj kr_gd S|tj krhgd S|tj krqgd S|tjkrzgd S|tjkrgdS|tjkrgdS|tjkrgdS|tjkrgdS|tjkrgdS|tjkrgdS|tjkrgdS|tjkrgdS|tjkrgdS|tjkrgdS|tjkrgdS|tjvrtd|dtd|dtt)a Retrieve a list of separators specific to the given language. Args: language (Language): The language for which to get the separators. Returns: List[str]: A list of separators appropriate for the specified language. ) class z void z int z float z double  if  for  while  switch  case r rFrGr )  func  var  const  type rTrUrWrXr rFrGr )rS public protected private  static rTrUrVrWrXr rFrGr )rSr]r^r_z internal z companion z fun  val rZrTrUrVz when rX else r rFrGr ) function r[ let rZrSrTrUrVrWrX default r rFrGr ) enum interface z namespace r\rSrcr[rdrZrTrUrVrWrXrer rFrGr ) rcrSrT foreach rV do rWrXr rFrGr ) z message z service rfz option  import z syntax r rFrGr )rS def z def r rFrGr )z =+ z -+ z \*+ z .. * r rFrGr ) rkrSrT unless rVrUriz begin z rescue r rFrGr )rkz defp z defmodule z defprotocol z defmacro z defmacrop rTrlrVrXz cond z with rUrir rFrGr ) z fn r[rdrTrVrUz loop  match r[r rFrGr )rSz object rkrarZrTrUrVrmrXr rFrGr )rYrS struct rfrTrUrVrirWrXr rFrGr ) z #{1,6} z``` z \*\*\*+ z ---+ z ___+ r rFrGr )z \\chapter{z \\section{z \\subsection{z \\subsubsection{z \\begin{enumerate}z \\begin{itemize}z \\begin{description}z \\begin{list}z \\begin{quote}z \\begin{quotation}z \\begin{verse}z \\begin{verbatim}z \\begin{align}z$$$rGr )zs