Every search engine, from a simple full-text search feature to a web-scale information retrieval system, begins with the same fundamental question: how do you turn raw text into searchable tokens? The text analysis pipeline is the chain of transformations that converts a document’s raw character stream into a sequence of normalized terms suitable for indexing and matching. Get this wrong, and no amount of ranking sophistication will save your search quality.
A typical text analysis pipeline has three core stages: tokenization (splitting text into individual words or tokens at word boundaries), filtering (removing stopwords—common words like “the”, “is”, “at” that carry little discriminative value), and stemming or lemmatization (reducing words to their root form so that “running”, “runs”, and “ran” all match the same stem). Additional stages may include lowercasing, accent folding, synonym expansion, and language detection.
As a concrete example, a high-performance text analysis pipeline implemented in Zig for use in a search engine’s indexing path illustrates these concepts well. It supports tokenization with Unicode-aware word boundary detection, Snowball stemming for 10+ languages, configurable stopword lists, and normalization (lowercasing, accent stripping). The pipeline is designed for zero-allocation steady-state operation, processing documents through a chain of analyzers that each transform the token stream in place.
Tokenization splits a character stream into discrete tokens (usually words). The simplest approach splits on whitespace, but real-world text requires handling punctuation, hyphens, apostrophes, URLs, email addresses, CJK characters (which lack spaces between words), and Unicode categories. The Unicode Text Segmentation standard (UAX #29) defines word boundary rules that handle most of these cases correctly.
The Unicode standard defines word boundary rules that account for apostrophes in contractions (don't → one token), numeric separators (3.14 → one token), and script-specific rules (e.g., no word boundaries between Katakana characters). A compliant tokenizer produces consistent, predictable token boundaries across all languages without language-specific heuristics.
Stopwords are high-frequency words that appear in almost every document and provide little value for distinguishing between documents. Common English stopwords include “the”, “and”, “is”, “in”, “to”. Removing them reduces index size (often by 25–40%) and improves query precision by preventing matches on ubiquitous terms.
Every language has its own set of stopwords. English stopword lists typically contain 150–300 words. German, with its many articles and prepositions, may have 400+. Chinese and Japanese don’t have traditional stopwords in the same sense, but have particles and connectors that serve a similar role. Using the wrong language’s stopword list can remove meaningful content words.
Stemming reduces words to a common root form (stem) so that morphological variants match the same index entry. The Snowball stemming framework (developed by Martin Porter, who also created the original Porter Stemmer) provides rule-based stemmers for 20+ languages. Snowball stemmers apply a sequence of suffix-stripping rules: “running” → “run”, “universities” → “univers”, “connections” → “connect”.
Stemming is a heuristic process that chops suffixes based on rules—it’s fast but sometimes produces non-word stems (“universities” → “univers”). Lemmatization uses a dictionary and morphological analysis to find the true base form (“universities” → “university”, “better” → “good”). Lemmatization is more accurate but significantly slower and requires language-specific dictionaries. For search engines, stemming is almost always preferred because the stems don’t need to be real words—they just need to be consistent between index time and query time.
Normalization ensures that surface-level differences don’t prevent matching. Common normalizations include: lowercasing (Hello → hello), accent/diacritic folding (café → cafe), Unicode normalization (NFC/NFKC to canonicalize equivalent character sequences), and character width folding (fullwidth → ASCII for CJK text).
The exact same analysis pipeline must be applied at both index time (when documents are stored) and query time (when search queries are processed). If the indexer stems “running” to “run” but the query analyzer doesn’t, the query “running” won’t match documents containing “running”. This symmetry requirement is the most common source of search bugs.
The analysis pipeline is a chain of composable stages. Each stage receives a stream of tokens and produces a (possibly modified) stream of tokens. The stages are:
The following tokenizer processes UTF-8 byte sequences and emits token boundaries based on Unicode word break rules. Here is a simplified word-boundary tokenizer:
const std = @import("std");
pub const Token = struct {
text: []const u8,
start: usize, // byte offset in original text
end: usize,
};
pub const Tokenizer = struct {
text: []const u8,
pos: usize,
pub fn init(text: []const u8) Tokenizer {
return .{ .text = text, .pos = 0 };
}
pub fn next(self: *Tokenizer) ?Token {
// Skip non-alphanumeric characters
while (self.pos < self.text.len) {
const c = self.text[self.pos];
if (std.ascii.isAlphanumeric(c)) break;
self.pos += 1;
}
if (self.pos >= self.text.len) return null;
const start = self.pos;
// Consume alphanumeric characters (word boundary)
while (self.pos < self.text.len) {
const c = self.text[self.pos];
if (!std.ascii.isAlphanumeric(c) and c != '\'') break;
self.pos += 1;
}
return Token{
.text = self.text[start..self.pos],
.start = start,
.end = self.pos,
};
}
};Snowball stemmers work by applying a series of conditional suffix-replacement rules. Each rule checks whether the remaining stem meets a minimum length or syllable count before applying. Here’s the conceptual flow for the English Snowball stemmer:
// Pseudocode for English Snowball stemmer (simplified)
fn stem(word: []u8) []u8 {
// Step 1a: Plurals and -ed/-ing
if endsWith(word, "sses") => replace with "ss"
if endsWith(word, "ies") => replace with "i"
if endsWith(word, "ss") => keep
if endsWith(word, "s") => remove
// Step 1b: Past tense and progressive
if endsWith(word, "eed") and measure > 0 => replace with "ee"
if endsWith(word, "ed") and containsVowel(stem) => remove "ed"
if endsWith(word, "ing") and containsVowel(stem) => remove "ing"
// Step 2: Derivational suffixes (-ational => -ate, -fulness => -ful, etc.)
// Step 3: More derivational suffixes (-icate => -ic, -ative => remove, etc.)
// Step 4: Remove -al, -ance, -ence, -er, -ic, -ment, -ness, -tion, etc.
// Step 5: Clean up (-e removal, double consonant reduction)
return word;
}Stopword filtering uses a hash set for O(1) lookup. The set is built once at initialization from a language-specific word list:
const std = @import("std");
pub const StopwordFilter = struct {
stopwords: std.StringHashMap(void),
pub fn init(allocator: std.mem.Allocator, words: []const []const u8) !StopwordFilter {
var map = std.StringHashMap(void).init(allocator);
for (words) |word| {
try map.put(word, {});
}
return .{ .stopwords = map };
}
pub fn isStopword(self: *const StopwordFilter, token: []const u8) bool {
return self.stopwords.contains(token);
}
pub fn deinit(self: *StopwordFilter) void {
self.stopwords.deinit();
}
};
// English stopwords (partial list)
const english_stopwords = [_][]const u8{
"the", "is", "at", "which", "on", "a", "an", "and", "or", "not",
"in", "to", "of", "for", "with", "as", "by", "from", "that", "this",
"it", "be", "are", "was", "were", "been", "being", "have", "has", "had",
};The complete analysis pipeline chains tokenization, lowercasing, stopword removal, and stemming. In Go (used for the query path):
package analysis
import (
"strings"
"unicode"
snowball "github.com/kljensen/snowball"
)
// Analyze runs the full text analysis pipeline on input text.
func Analyze(text string, lang string, stopwords map[string]bool) []string {
// Step 1: Tokenize on word boundaries
tokens := tokenize(text)
// Step 2: Normalize (lowercase)
for i, t := range tokens {
tokens[i] = strings.ToLower(t)
}
// Step 3: Remove stopwords
filtered := tokens[:0]
for _, t := range tokens {
if !stopwords[t] {
filtered = append(filtered, t)
}
}
// Step 4: Stem each token
stemmed := make([]string, len(filtered))
for i, t := range filtered {
s, err := snowball.Stem(t, lang, true)
if err != nil {
stemmed[i] = t // fall back to unstemmed
} else {
stemmed[i] = s
}
}
return stemmed
}
func tokenize(text string) []string {
var tokens []string
start := -1
for i, r := range text {
if unicode.IsLetter(r) || unicode.IsDigit(r) {
if start == -1 {
start = i
}
} else {
if start != -1 {
tokens = append(tokens, text[start:i])
start = -1
}
}
}
if start != -1 {
tokens = append(tokens, text[start:])
}
return tokens
}A typical search engine runs every document through the text analysis pipeline before adding terms to the inverted index. The analyzer is configured per-field: a “title” field might use a light stemmer and keep more stopwords (to support exact phrase matching), while a “body” field uses aggressive stemming and full stopword removal. At query time, the same analyzer chain processes the query text to ensure symmetric matching.
For systems that index content in multiple languages, the analyzer must be language-aware. A common approach is to detect the document’s language (using character n-gram statistics or a library like CLD3), then select the appropriate Snowball stemmer and stopword list. A Zig-based search engine can support 10+ languages by embedding Snowball stemmers compiled from the canonical Snowball source into the binary.
Autocomplete systems use a modified analysis pipeline: they tokenize and normalize but typically skip stemming (users expect to see their exact prefix, not a stemmed version). Edge n-grams—generating prefixes of each token (“search” → “s”, “se”, “sea”, “sear”, “searc”, “search”)—are indexed so that prefix queries match without wildcard scans.
Log search systems (like ELK or Loki) use text analysis pipelines tuned for structured log formats. Tokenizers handle camelCase splitting, dot-separated identifiers (com.example.MyClass), and IP addresses as single tokens. Stopword lists are replaced with domain-specific filters that remove high-frequency log boilerplate.
A common pattern is a registry that maps analyzer names to configured pipeline instances. This allows index schemas to reference analyzers by name:
package analysis
// AnalyzerRegistry maps analyzer names to configured pipeline instances.
type AnalyzerRegistry struct {
analyzers map[string]*Analyzer
}
type Analyzer struct {
Tokenizer TokenizerFunc
Filters []TokenFilterFunc
Stopwords map[string]bool
StemLanguage string
}
type TokenizerFunc func(string) []string
type TokenFilterFunc func([]string) []string
func NewRegistry() *AnalyzerRegistry {
r := &AnalyzerRegistry{analyzers: make(map[string]*Analyzer)}
// Register default analyzers
r.Register("english", &Analyzer{
Tokenizer: UnicodeTokenizer,
Filters: []TokenFilterFunc{LowercaseFilter, StopwordFilter(EnglishStopwords)},
StemLanguage: "english",
})
r.Register("german", &Analyzer{
Tokenizer: UnicodeTokenizer,
Filters: []TokenFilterFunc{LowercaseFilter, StopwordFilter(GermanStopwords)},
StemLanguage: "german",
})
r.Register("keyword", &Analyzer{
Tokenizer: KeywordTokenizer, // treats entire input as one token
Filters: []TokenFilterFunc{LowercaseFilter},
})
return r
}
func (r *AnalyzerRegistry) Register(name string, a *Analyzer) {
r.analyzers[name] = a
}
func (r *AnalyzerRegistry) Get(name string) *Analyzer {
return r.analyzers[name]
}In hot paths (indexing millions of documents), allocating token slices is expensive. The Zig implementation uses an iterator pattern that yields token views into the original text without allocation:
// The Tokenizer.next() method shown earlier returns slices into the
// original text buffer - no allocation needed. The caller processes
// each token inline:
var tokenizer = Tokenizer.init(document_text);
while (tokenizer.next()) |token| {
const lower = toLowerInPlace(token.text);
if (!stopwords.isStopword(lower)) {
const stem = stemmer.stem(lower);
index.addTerm(stem, doc_id, token.start);
}
}In a search engine’s indexing pipeline, text analysis typically consumes 40–60% of total CPU time. The tokenizer runs over every byte of every document; the stemmer applies complex suffix rules to every token. Optimizing the analysis pipeline (SIMD character classification, precomputed stemmer tables, arena allocation) has an outsized impact on overall indexing throughput.
The most common search bug: using a different analyzer (or analyzer configuration) at index time vs. query time. If you add a new stopword to the index analyzer but not the query analyzer, queries containing that word will search for a term that no longer exists in the index. Always configure analyzers centrally and reference them by name from both index and query code paths.
Stemming can conflate unrelated words. The Porter stemmer maps both “universe” and “university” to “univers”, causing false matches. The Snowball stemmer is less aggressive but still imperfect. For domains where precision matters (legal, medical), consider lighter stemming or lemmatization.
If a user searches for "to be or not to be", removing stopwords leaves nothing. Phrase queries and exact-match queries often need stopwords preserved. Many search engines apply stopword removal only to bag-of-words queries, not phrase queries.
The character “é” can be represented as a single codepoint (U+00E9) or as two codepoints (U+0065 + U+0301). If your tokenizer doesn’t normalize to NFC/NFKC, the same visual character produces different byte sequences, leading to missed matches. Always normalize Unicode before tokenization.
Chinese, Japanese, and Korean text doesn’t use spaces between words. A whitespace tokenizer produces one giant token per paragraph. CJK text requires either dictionary-based segmentation (like jieba for Chinese or MeCab for Japanese) or n-gram tokenization (bigrams of characters). Using a Latin-oriented tokenizer on CJK text silently produces unusable indexes.
Overly aggressive stopword lists remove words that users actually search for. For example, removing “not” changes the meaning of “not working” to “working”. Start with a conservative stopword list (50–100 words) and expand only when index size or query quality justifies it.