Semantic Chunking

Overview

Semantic chunking is a text splitting strategy that divides documents based on meaning rather than arbitrary size limits. Instead of cutting text every N characters or tokens, semantic chunking analyzes the relationships between sentences and creates boundaries where the topic or meaning shifts. The result is chunks that contain complete, coherent thoughts — each one a self-contained unit of information.

This approach directly addresses the primary weakness of fixed-size chunking: severed context. When a fixed-size chunk cuts a paragraph in half, the resulting fragments embed poorly because they represent incomplete ideas. Semantic chunking produces variable-sized chunks that respect natural language boundaries, leading to higher-quality embeddings and more accurate retrieval in RAG systems.

The tradeoff is computational cost. Semantic chunking requires an embedding model to generate sentence-level vectors and then performs similarity calculations to detect topic shifts. For a 10,000-document corpus, this can mean millions of additional embedding calls during preprocessing. Understanding when this cost is justified — and when simpler methods suffice — is a critical decision in pipeline design.

How It Works

The Core Algorithm

Semantic chunking follows a three-step process:

1. Sentence splitting: The input text is divided into individual sentences using NLP sentence boundary detection (e.g., spaCy, NLTK, or regex-based splitters).
2. Embedding generation: Each sentence is independently embedded into a dense vector using an embedding model (e.g., text-embedding-3-small, all-MiniLM-L6-v2).
3. Similarity-based grouping: Consecutive sentence embeddings are compared using cosine similarity. When the similarity between adjacent sentences drops below a threshold, a chunk boundary is created.

Sliding Window Approach

A more sophisticated variant uses a sliding window of sentences rather than comparing only adjacent pairs. For example, with a window size of 3, the algorithm computes the average embedding of sentences [i-1, i, i+1] and compares it to [i, i+1, i+2]. This smooths out noise from individual sentence variations and produces more stable breakpoints:

import numpy as np
from sentence_transformers import SentenceTransformer

def semantic_chunk(text, threshold=0.75, window_size=1):
    # Step 1: Split into sentences
    sentences = split_into_sentences(text)

    # Step 2: Embed each sentence
    model = SentenceTransformer('all-MiniLM-L6-v2')
    embeddings = model.encode(sentences)

    # Step 3: Find breakpoints via cosine similarity
    chunks = []
    current_chunk = [sentences[0]]

    for i in range(1, len(sentences)):
        # Compare embedding of current sentence to previous
        sim = cosine_similarity(
            embeddings[i].reshape(1, -1),
            embeddings[i - 1].reshape(1, -1)
        )[0][0]

        if sim < threshold:
            # Semantic shift detected — start new chunk
            chunks.append(" ".join(current_chunk))
            current_chunk = [sentences[i]]
        else:
            current_chunk.append(sentences[i])

    # Don't forget the last chunk
    if current_chunk:
        chunks.append(" ".join(current_chunk))

    return chunks

Percentile-Based Thresholding

Setting an absolute similarity threshold is fragile — what works for one document may fail for another. A more robust approach computes all pairwise adjacent similarities first, then sets the threshold as a percentile. For example, breaking at the bottom 10th percentile of similarity scores means you create boundaries at the 10% most dramatic topic shifts in each specific document:

# Compute all adjacent similarities
similarities = []
for i in range(1, len(embeddings)):
    sim = cosine_similarity(embeddings[i], embeddings[i-1])
    similarities.append(sim)

# Set threshold at 10th percentile (break at biggest shifts)
threshold = np.percentile(similarities, 10)

# Use this document-specific threshold for breakpoint detection

LangChain SemanticChunker

LangChain provides a built-in SemanticChunker that implements this pattern with configurable breakpoint types:

from langchain_experimental.text_splitter import SemanticChunker
from langchain_openai import OpenAIEmbeddings

chunker = SemanticChunker(
    OpenAIEmbeddings(),
    breakpoint_threshold_type="percentile",  # or "standard_deviation", "interquartile"
    breakpoint_threshold_amount=10           # bottom 10% similarity = breakpoints
)

chunks = chunker.split_text(document)

Variable Chunk Sizes

A defining characteristic of semantic chunking is that chunk sizes vary significantly. A dense technical paragraph might produce a 50-token chunk, while a long narrative section on a single topic might yield a 600-token chunk. This variability is both a strength (chunks are semantically coherent) and a challenge (you may need to enforce minimum and maximum size constraints to avoid very small or very large chunks that cause problems downstream).

Practical Applications

When Semantic Chunking Excels

• Multi-topic documents: Research papers, reports, and articles that cover multiple distinct topics benefit enormously from semantic chunking. Each section gets its own chunk, so retrieval returns the relevant section rather than a fragment spanning two topics.
• Conversational data: Customer support transcripts, meeting notes, and chat logs where topics shift frequently. Semantic chunking isolates each discussion topic into its own retrievable unit.
• High-precision retrieval: When retrieval quality is critical and preprocessing cost is acceptable — medical knowledge bases, legal document search, financial analysis — semantic chunking’s superior chunk coherence translates directly to better answers.

Performance Considerations

Semantic chunking is significantly more expensive than fixed-size chunking at preprocessing time:

• Embedding calls: Every sentence requires an embedding API call. A 2000-word document with ~100 sentences means 100 embedding calls just for chunking — before the final chunk embeddings used for retrieval.
• Latency: For a 10,000-document corpus, semantic chunking can take 10-50x longer than fixed-size chunking depending on the embedding model and batch sizes.
• Cost: With paid embedding APIs, the per-sentence embedding cost adds up. For OpenAI’s text-embedding-3-small at ~$0.02/1M tokens, a 10,000-document corpus might cost $5-15 extra for the chunking step alone.

However, these are one-time preprocessing costs. If improved retrieval quality leads to better user-facing answers, the investment pays for itself many times over.

Combining with Size Constraints

In production systems, pure semantic chunking is often combined with size guards. After semantic breakpoints are identified, any chunk exceeding a maximum size (e.g., 512 tokens) is further split using fixed-size or recursive chunking, while chunks below a minimum size (e.g., 50 tokens) are merged with their neighbors. This hybrid approach preserves semantic coherence while respecting practical embedding model limits.

Key Takeaways

• Semantic chunking splits text at points where the meaning shifts, using embedding similarity between consecutive sentences to detect topic boundaries.
• The three-step process is: split into sentences → embed each sentence → group by similarity, breaking at low-similarity points.
• Percentile-based thresholds (e.g., break at the bottom 10% of similarity scores) are more robust than absolute thresholds across different document types.
• Produces variable-sized chunks that are semantically coherent — each chunk contains a complete thought or topic.
• Computationally expensive: requires per-sentence embedding calls, making it 10-50x slower than fixed-size chunking at preprocessing time.
• Best suited for multi-topic documents, high-precision retrieval scenarios, and cases where preprocessing cost is acceptable.
• In production, combine with size guards (min/max token limits) to handle edge cases of very small or very large semantic chunks.