o Zh@s<ddlmZddlmZmZddlmZGdddeZdS)) annotations)AnyList) TextSplittercs8eZdZdZ  ddddfddZdddZZS)NLTKTextSplitterz"Splitting text using NLTK package. englishF)use_span_tokenize separatorstrlanguager boolkwargsrreturnNonec stjdi|||_||_||_|jr|jdkrtdz|jr1ddlm}||j|_WdSddlm }||_WdSt yGt dw) zInitialize the NLTK splitter.z6When use_span_tokenize is True, separator should be ''r)_get_punkt_tokenizer) sent_tokenizezANLTK is not installed, please install it with `pip install nltk`.N) super__init__ _separator _language_use_span_tokenize ValueErrorZ nltk.tokenizer _tokenizerr ImportError)selfr r r rrr __class__rT/var/www/html/lang_env/lib/python3.10/site-packages/langchain_text_splitters/nltk.pyr s"     zNLTKTextSplitter.__init__text List[str]c Cs|jr=t|j|}g}t|D]*\}\}}|dkr0||dd}||||||}n|||}||qn|j||jd}|||jS)z&Split incoming text and return chunks.r)r ) rlistrZ span_tokenize enumerateappendrZ _merge_splitsr) rr!spansZsplitsistartendZprev_endZsentencerrr split_text(s  zNLTKTextSplitter.split_text)rr) r r r r r r rrrr)r!r rr")__name__ __module__ __qualname____doc__rr+ __classcell__rrrr rsrN) __future__rtypingrrZlangchain_text_splitters.baserrrrrr s