What are various text splitters and their differences?

In Langchain and other text-processing frameworks for Large Language Models (LLMs), different text-splitting strategies are available to handle large documents effectively and avoid exceeding token limits. Here are some options for text splitting and how they differ:

1. RecursiveCharacterTextSplitter

How it works: This splitter recursively breaks down large documents into smaller chunks using a hierarchy of delimiters like paragraphs, sentences, and words.

Pros:

Preserves logical structure (e.g., paragraphs).

Handles large chunks while retaining context.

Cons:

Slower for very large documents due to recursion.

Key method:

RecursiveCharacterTextSplitter.from_tiktoken_encoder: Uses token-based splitting (specific to token encoding methods like OpenAI’s tiktoken).

2. CharacterTextSplitter

How it works: This simpler splitter divides text based on a single character or set of characters (like space, newline).

Pros:

Fast and efficient for straightforward splitting.

Cons:

Can split content mid-sentence or mid-paragraph, which might disrupt context.

Key method:

Splits based on specific delimiters like newline or any character of choice.

Use Case: Fast processing, where document structure is not as important.

3. N-gram Text Splitters

How it works: Splits documents into smaller chunks based on N-grams (groups of n words or tokens).

Pros:

Good for splitting while maintaining word-level granularity.

Cons:

Might lose structural integrity (i.e., paragraphs, sentences).

Use Case: Ideal for small token batches, especially for keyword-based searches or training LLMs that need sliding window approaches.

4. SentenceTextSplitter

How it works: Splits text based on sentence boundaries using natural language processing (NLP) techniques.

Pros:

Maintains logical sentence-level coherence.

Great for applications where each sentence matters (e.g., summarization, question-answering).

Cons:

Might generate larger chunks, leading to token overflow if not used in combination with other splitters.

Use Case: Tasks that require sentence-level processing, like summarization.

5. TokenTextSplitter

How it works: Splits based on tokens (e.g., word tokens or byte-pair encodings) instead of characters.

Pros:

Precise control over token limits, perfect for LLMs like GPT that have token-based inputs.

Cons:

May cut off mid-word or mid-sentence if token boundaries don’t align with logical text structures.

Use Case: Managing token limits when interacting with models that operate on tokens.

6. Document Splitters (Custom)

How it works: Custom splitters that use domain-specific knowledge (e.g., splitting based on document sections or code blocks).

Pros:

Allows customization based on specific document formats (e.g., splitting code by function).

Cons:

Requires custom logic, making implementation more complex.

Use Case: Domain-specific documents like programming code, academic papers, or medical reports.

Differences Between These Methods:

Granularity: Some splitters (like SentenceTextSplitter) work on a higher granularity (sentences), while others (TokenTextSplitter, CharacterTextSplitter) can split at lower levels (characters, tokens).

Structure Preservation: Recursive splitters preserve the document structure better, while simpler methods (like CharacterTextSplitter) may sacrifice coherence for speed.

Speed: Simpler splitters like CharacterTextSplitter are faster but can disrupt text flow. Recursive methods are slower but more robust for maintaining context.

Use Cases: RecursiveCharacterTextSplitter is better for documents requiring context retention, while TokenTextSplitter is more useful for precise control over token consumption.

Depending on your task, you can choose a splitter that balances speed, context preservation, and granularity of chunks.