Full-text search is the foundational retrieval paradigm that has powered search engines for decades. Unlike semantic search, which operates on meaning, full-text search works on vocabulary overlap—it finds documents that contain the exact words (or stems of words) present in the query. The dominant scoring algorithm is BM25 (Best Match 25), which ranks documents by how frequently and uniquely query terms appear in them.
In AI engineering, full-text search remains essential because semantic search alone has significant blind spots. Embedding models struggle with exact identifiers like product codes (SKU-4829-X), serial numbers, acronyms, and proper nouns that have no inherent semantic content. BM25 excels at these exact-match scenarios because it operates on literal token overlap. This is why production RAG systems almost always combine semantic and full-text search in a hybrid approach.
Bleve is a full-featured, open-source search and indexing library written in Go that implements BM25 scoring along with rich text analysis capabilities. It’s particularly relevant because it’s the search engine embedded in systems like Couchbase and is designed for embedding directly into Go applications—a common pattern in AI infrastructure built with Go backends.
Full-text search is built on the inverted index—a data structure that maps every unique term to the list of documents containing it. When you index a document, the system tokenizes the text, applies analyzers (lowercasing, stemming, stop word removal), and stores the resulting terms in the index with positional information.
// Conceptual inverted index structure
"deployment" → [doc_1 (pos: 5, 42), doc_7 (pos: 12), doc_15 (pos: 3, 88)]
"kubernetes" → [doc_1 (pos: 8), doc_3 (pos: 1, 22), doc_7 (pos: 15)]
"pipeline" → [doc_1 (pos: 6), doc_15 (pos: 4)]At query time, the system looks up each query term in the inverted index, finds the matching documents, and then scores them using BM25.
BM25 scores a document against a query by combining three factors for each query term:
score(D, Q) = Σ IDF(qi) × [ f(qi, D) × (k1 + 1) ] / [ f(qi, D) + k1 × (1 - b + b × |D| / avgdl) ]
Where f(qi, D) is term frequency of query term qi in document D, |D| is document length, avgdl is average document length, k1 controls term frequency saturation (default: 1.2), and b controls length normalization (default: 0.75).
IDF measures how rare a term is across the entire corpus. Common words like “the” or “is” appear in nearly every document and receive a very low IDF score. Rare, distinctive terms like “Kubernetes” or “autosharding” receive high IDF scores. This ensures that matching on a distinctive term contributes more to the relevance score than matching on a ubiquitous one.
The formula is: IDF(q) = ln((N - n(q) + 0.5) / (n(q) + 0.5) + 1), where N is the total number of documents and n(q) is the number of documents containing term q.
Term frequency (TF) counts how often a query term appears in a document. But raw TF is misleading—a document mentioning “database” 100 times isn’t 100x more relevant than one mentioning it once. BM25 applies a saturation curve controlled by parameter k1: as TF increases, the score contribution grows rapidly at first, then flattens out. With the default k1 = 1.2, the score is already at ~70% of its maximum after just 3 occurrences.
Longer documents naturally contain more term occurrences, which would unfairly boost their scores. BM25 normalizes by comparing a document’s length to the average length across the corpus. The parameter b (default: 0.75) controls how aggressively this normalization is applied. At b = 0, there’s no length normalization; at b = 1, the penalty for being longer than average is fully applied.
Before terms enter the index, they pass through an analysis pipeline:
Bleve is a pure-Go search library that provides BM25 scoring, rich text analysis, and multiple index backends. It’s designed to be embedded directly into Go applications without requiring an external search service.
// Bleve: create index, add document, and search in Go
package main
import "github.com/blevesearch/bleve/v2"
func main() {
// Create a new index
mapping := bleve.NewIndexMapping()
index, _ := bleve.New("myindex.bleve", mapping)
// Index a document
doc := map[string]interface{}{
"title": "CI/CD Troubleshooting Guide",
"body": "Common deployment pipeline failures and fixes...",
}
index.Index("doc-1", doc)
// Search
query := bleve.NewMatchQuery("deployment pipeline")
req := bleve.NewSearchRequest(query)
results, _ := index.Search(req)
// results.Hits contains ranked matches
}Key Bleve capabilities include:
In production RAG systems, BM25 runs alongside semantic search. The query “error code E-4829 in deployment” benefits from BM25 finding exact matches on “E-4829” while semantic search finds conceptually related troubleshooting documents. Results from both systems are merged using fusion algorithms (RRF or RSF) for final ranking.
Support systems, inventory databases, and technical documentation are full of identifiers—product SKUs, error codes, version numbers, API endpoint paths. BM25 handles these perfectly because it operates on literal token matching. Searching for v2.14.3-beta returns exactly the documents containing that string.
Bleve is commonly used in Go-based microservices that need search capabilities without the operational overhead of running Elasticsearch or Solr. A document management service can embed Bleve directly, using the local filesystem for index storage and avoiding network latency for search queries.
k1 (default 1.2) and b (default 0.75) control term frequency saturation and length normalization respectively.