Recursive Chunking

Overview

Recursive chunking is the industry-standard method for splitting documents in RAG pipelines. It combines the predictable size control of fixed-size chunking with structural awareness by using a hierarchy of separators — trying the most significant boundaries first (double newlines for paragraphs) and progressively falling back to finer separators (single newlines, spaces, individual characters) only when chunks exceed the target size. This approach is the default in LangChain and is recommended by most RAG framework documentation.

The key insight behind recursive chunking is that text has a natural hierarchy of structure. Double newlines typically separate paragraphs, which are the strongest semantic boundaries in unformatted text. Single newlines separate lines within a paragraph. Spaces separate words. By respecting this hierarchy — splitting at paragraphs first, then sentences, then words — recursive chunking keeps semantically related text together as much as possible while still enforcing a maximum chunk size.

Unlike semantic chunking, recursive chunking requires no embedding model and no NLP processing. It operates purely on string separators, making it nearly as fast as fixed-size chunking but with significantly better output quality. For most RAG applications, recursive chunking is the right default choice — it’s the sweet spot between simplicity and semantic coherence.

How It Works

The Separator Hierarchy

Recursive chunking is parameterized by an ordered list of separators, tried from most significant to least significant. The default separator list for generic text is:

1. "\n\n" — Double newline (paragraph boundary)
2. "\n" — Single newline (line boundary)
3. " " — Space (word boundary)
4. "" — Empty string (character boundary, last resort)

Step-by-Step Example

Consider a document with the following structure (sizes in tokens):

Paragraph 1 (150 tokens)
\n\n
Paragraph 2 (400 tokens)    ← exceeds 256-token limit
\n\n
Paragraph 3 (200 tokens)
\n\n
Paragraph 4 (100 tokens)

With chunk_size=256 and chunk_overlap=20:

1. Split on "\n\n": Produces 4 segments. Paragraphs 1, 3, and 4 are under 256 tokens — they become chunks as-is.
2. Paragraph 2 exceeds 256 tokens: Recursively split on "\n" (line breaks). If still too large, split on " " (spaces).
3. Merge small chunks: Paragraphs 3 (200 tokens) and 4 (100 tokens) might be combined into a single 300-token chunk if that’s under the limit, or Paragraph 4 might be merged with overlap from Paragraph 3.
4. Apply overlap: The last 20 tokens of each chunk are repeated at the start of the next chunk.

LangChain RecursiveCharacterTextSplitter

The RecursiveCharacterTextSplitter is LangChain’s default and most recommended text splitter:

from langchain.text_splitter import RecursiveCharacterTextSplitter

splitter = RecursiveCharacterTextSplitter(
    chunk_size=1000,         # max characters per chunk
    chunk_overlap=200,       # overlap between chunks
    separators=["\n\n", "\n", " ", ""],  # default hierarchy
    length_function=len      # or use a token counter
)

chunks = splitter.split_text(document)

# Each chunk will be ≤ 1000 characters
# Paragraph boundaries preserved where possible
# Long paragraphs split at line or word boundaries

Token-Aware Recursive Splitting

For production RAG systems, you want chunk sizes measured in tokens, not characters, since embedding models have token limits. LangChain supports this with a custom length function:

import tiktoken

# Create a token-counting length function
enc = tiktoken.encoding_for_model("text-embedding-3-small")
def token_length(text):
    return len(enc.encode(text))

splitter = RecursiveCharacterTextSplitter(
    chunk_size=256,           # max tokens per chunk
    chunk_overlap=25,         # token overlap
    separators=["\n\n", "\n", ". ", " ", ""],
    length_function=token_length
)

chunks = splitter.split_text(document)

Note the addition of ". " (period-space) to the separator list. This splits at sentence boundaries before falling back to word boundaries, which is a common enhancement for prose-heavy documents.

Language-Specific Separators

One of recursive chunking’s greatest strengths is its adaptability to different content types through custom separator lists. LangChain provides built-in language-aware separators for programming languages:

from langchain.text_splitter import RecursiveCharacterTextSplitter, Language

# Python code splitting
python_splitter = RecursiveCharacterTextSplitter.from_language(
    language=Language.PYTHON,
    chunk_size=1000,
    chunk_overlap=100
)
# Uses separators like: ["\nclass ", "\ndef ", "\n\n", "\n", " ", ""]
# Keeps classes and functions intact when possible

# Markdown splitting
md_splitter = RecursiveCharacterTextSplitter.from_language(
    language=Language.MARKDOWN,
    chunk_size=1000,
    chunk_overlap=100
)
# Uses separators like: ["\n# ", "\n## ", "\n### ", "\n\n", "\n", " ", ""]
# Respects heading hierarchy

Custom Separator Lists

For domain-specific documents, crafting a custom separator list can dramatically improve chunk quality:

• Legal documents: ["\nArticle ", "\nSection ", "\n\n", "\n", ". ", " ", ""]
• HTML content: ["

Practical Applications

Why Recursive Chunking is the Default

Recursive chunking occupies the sweet spot in the chunking strategy landscape:

• Speed: No embedding model calls, no NLP processing. Pure string operations. Nearly as fast as fixed-size chunking.
• Quality: Respects document structure. Paragraphs stay together. Sentences aren’t severed mid-thought. Significantly better retrieval quality than fixed-size chunking.
• Predictability: Chunk sizes are bounded by chunk_size. Unlike semantic chunking, you won’t get a 2000-token chunk that exceeds your embedding model’s limit.
• Adaptability: Custom separator lists let you handle any document format without changing the core algorithm.

Benchmarks: Recursive vs Fixed-size vs Semantic

In practice, teams typically observe the following patterns when comparing chunking strategies on RAG retrieval benchmarks:

• Fixed-size chunking: Baseline. Fast preprocessing. Adequate retrieval for homogeneous text.
• Recursive chunking: 5-15% improvement in retrieval precision over fixed-size, with negligible preprocessing overhead. The go-to default.
• Semantic chunking: 2-8% improvement over recursive chunking on multi-topic documents, but 10-50x slower preprocessing. Worth it only for high-value, heterogeneous corpora.

Production Configuration

A battle-tested production configuration for a general-purpose RAG pipeline:

# Production recursive chunking config
splitter = RecursiveCharacterTextSplitter(
    chunk_size=512,            # tokens (not chars) — matches most embedding model limits
    chunk_overlap=50,          # ~10% overlap
    separators=[
        "\n\n",                # paragraph boundaries
        "\n",                  # line boundaries
        ". ",                  # sentence boundaries
        "? ",                  # question boundaries
        "! ",                  # exclamation boundaries
        "; ",                  # clause boundaries
        ", ",                  # phrase boundaries
        " ",                   # word boundaries
        ""                     # character fallback
    ],
    length_function=token_length,
    strip_whitespace=True
)

Combining with Metadata

In production RAG systems, chunks carry metadata alongside their text content. Recursive chunking integrates well with document-level metadata propagation:

from langchain.text_splitter import RecursiveCharacterTextSplitter
from langchain.schema import Document

# Split documents while preserving metadata
docs = [Document(page_content=text, metadata={"source": "manual_v3.pdf", "page": 12})]
splitter = RecursiveCharacterTextSplitter(chunk_size=500, chunk_overlap=50)
chunks = splitter.split_documents(docs)

# Each chunk inherits the parent document's metadata
# chunks[0].metadata == {"source": "manual_v3.pdf", "page": 12}

Key Takeaways

• Recursive chunking uses an ordered list of separators — trying the most significant boundaries first and falling back to finer separators only when chunks are too large.
• The default separator hierarchy is ["\n\n", "\n", " ", ""], which tries paragraphs → lines → words → characters.
• It’s the industry standard and LangChain default because it balances speed, quality, and predictability better than any other strategy.
• No embedding model or NLP processing required — operates purely on string separators, making it nearly as fast as fixed-size chunking.
• Custom separator lists adapt the algorithm to any document format: code, Markdown, legal documents, HTML, and more.
• Use token-based length functions (not character-based) in production to ensure chunks respect embedding model token limits.
• Typically delivers 5-15% better retrieval precision than fixed-size chunking with negligible preprocessing overhead.
• Add sentence-level separators like ". " and "? " for prose-heavy documents to improve split quality.