o Zh@sLddlmZddlmZmZmZmZddlmZm Z m Z GdddeZ dS)) annotations)AnyListOptionalcast) TextSplitter Tokenizersplit_text_on_tokenscs`eZdZUdZ   ddfdd Zd ddZd!ddZd"ddZdZde d<d#ddZ Z S)$%SentenceTransformersTokenTextSplitterz8Splitting text to tokens using sentence model tokenizer.2'sentence-transformers/all-mpnet-base-v2N chunk_overlapint model_namestrtokens_per_chunk Optional[int]kwargsrreturnNonec sntjdi|d|izddlm}Wn ty tdw||_||j|_|jj|_|j|ddS)zCreate a new TextSplitter.r r)SentenceTransformerzCould not import sentence_transformers python package. This is needed in order to for SentenceTransformersTokenTextSplitter. Please install it with `pip install sentence-transformers`.)rN) super__init__Zsentence_transformersr ImportErrorr_model tokenizer_initialize_chunk_configuration)selfr rrrr __class__re/var/www/html/lang_env/lib/python3.10/site-packages/langchain_text_splitters/sentence_transformers.pyr s   z.SentenceTransformersTokenTextSplitter.__init__cCsZtt|jj|_|dur|j|_n||_|j|jkr+td|jd|jd|jddS)NzThe token limit of the models 'z' is: z. Argument tokens_per_chunk=z > maximum token limit.)rrrZmax_seq_lengthZmaximum_tokens_per_chunkr ValueErrorr)rrrrr!r#s   zESentenceTransformersTokenTextSplitter._initialize_chunk_configurationtext List[str]cs2d fdd }tjjjj|d}t||dS) aSplits the input text into smaller components by splitting text on tokens. This method encodes the input text using a private `_encode` method, then strips the start and stop token IDs from the encoded result. It returns the processed segments as a list of strings. Args: text (str): The input text to be split. Returns: List[str]: A list of string components derived from the input text after encoding and processing. r#rr List[int]cs|ddS)N)_encode)r#rrr!%encode_strip_start_and_stop_token_idsDsz_SentenceTransformersTokenTextSplitter.split_text..encode_strip_start_and_stop_token_ids)r rdecodeencode)r#rNr#rrr%)rZ_chunk_overlaprrr+r )rr#r*rrr)r! split_text5s z0SentenceTransformersTokenTextSplitter.split_textcCst||S)ayCounts the number of tokens in the given text. This method encodes the input text using a private `_encode` method and calculates the total number of tokens in the encoded result. Args: text (str): The input text for which the token count is calculated. Returns: int: The number of tokens in the encoded text. )lenr()rr#rrr! count_tokensPs z2SentenceTransformersTokenTextSplitter.count_tokensl _max_length_equal_32_bit_integerr%cCs|jj||jdd}|S)NZdo_not_truncate) max_lengthZ truncation)rr,r1)rr#Z&token_ids_with_start_and_end_token_idsrrr!r(`s z-SentenceTransformersTokenTextSplitter._encode)r r N) r rrrrrrrrr)rrrr)r#rrr$)r#rrrr-) __name__ __module__ __qualname____doc__rrr.r0r1__annotations__r( __classcell__rrrr!r s     r N) __future__rtypingrrrrZlangchain_text_splitters.baserrr r rrrr!s