Semantic search is a retrieval paradigm that finds documents based on meaning rather than exact keyword matches. Instead of relying on word overlap between a query and a document, semantic search converts both into dense vector embeddings—numerical representations in a high-dimensional space—and measures the distance between them. Documents whose vectors are closest to the query vector are returned as the most relevant results.
This is a foundational capability for AI engineering. Traditional keyword search fails when users phrase queries differently from how documents are written. A user searching for “how to fix a broken deployment pipeline” would miss a document titled “CI/CD troubleshooting guide” in a keyword system, but a semantic search system understands these are about the same concept and returns it at the top.
Semantic search is the core retrieval mechanism behind Retrieval-Augmented Generation (RAG) systems. When an LLM needs to ground its response in external data, semantic search is typically the first step: it finds the most contextually relevant chunks to inject into the model’s context window. Getting this step right is critical—if the retriever returns irrelevant results, even the best LLM will produce poor answers.
An embedding model (also called an encoder) converts text into a fixed-length numerical vector, typically between 256 and 1536 dimensions. Popular models include OpenAI’s text-embedding-3-small (1536 dimensions), Cohere’s embed-v3, and open-source models like all-MiniLM-L6-v2 from Sentence Transformers (384 dimensions).
The key property is that semantically similar texts produce vectors that are close together in the embedding space. “Car” and “vehicle” end up near each other, while “car” and “banana” are far apart. Each dimension captures a learned semantic feature—not a human-interpretable one, but a statistical pattern the model learned during training.
Semantic search uses dense vectors where most values are non-zero (e.g., 768 floating-point numbers). This contrasts with sparse vectors used in BM25/TF-IDF, where vectors have tens of thousands of dimensions but most are zero. Dense vectors capture meaning; sparse vectors capture vocabulary overlap.
During ingestion, every document (or chunk) is passed through the embedding model and the resulting vector is stored in a vector database or index. This is a one-time cost per document. For a corpus of 1 million chunks at 768 dimensions using float32, you need approximately 3 GB of storage just for the vectors.
# Pseudocode: indexing documents
from sentence_transformers import SentenceTransformer
model = SentenceTransformer("all-MiniLM-L6-v2")
documents = ["CI/CD troubleshooting guide...", "Kubernetes scaling patterns...", ...]
embeddings = model.encode(documents) # Shape: (N, 384)
# Store in vector database
vector_db.upsert(ids=doc_ids, vectors=embeddings, metadata=doc_metadata)At query time, the user’s query is encoded with the same embedding model to produce a query vector. The system then computes the similarity between the query vector and every indexed document vector, returning the top-k most similar results.
The most common similarity metric for text embeddings. It measures the cosine of the angle between two vectors, producing a score from -1 (opposite) to 1 (identical). For normalized vectors (unit length), cosine similarity equals the dot product, making computation extremely fast. Most modern embedding models output normalized vectors by default.
Computing exact similarity against every vector is O(n)—too slow for large corpora. Production systems use Approximate Nearest Neighbor algorithms like HNSW (Hierarchical Navigable Small World) to achieve sub-millisecond search across millions of vectors with 95-99% recall. HNSW builds a multi-layer graph where each node connects to its nearest neighbors, allowing the search to “hop” through the graph efficiently.
For datasets under 10 million vectors, HNSW is the industry default. It delivers the best balance of recall, latency, and memory usage. Typical benchmarks show 1-5ms query latency at 98%+ recall across 1M vectors.
Semantic search systems use a bi-encoder architecture: query and document are encoded independently by the same model. This is what makes it scalable—documents can be pre-encoded offline, and only the query needs encoding at search time (typically 5-20ms). The tradeoff is that the model never sees query and document together, which limits its ability to capture fine-grained interactions. This is why rerankers (cross-encoders) are often used as a second stage.
The most common use case in AI engineering. When a user asks a question, the system encodes the query, retrieves the top-k most relevant chunks from the vector database, and injects them into the LLM’s prompt. Semantic search ensures the model receives contextually appropriate information, even when the user’s phrasing doesn’t match the source material verbatim.
Enterprise search across heterogeneous documents—Confluence pages, PDFs, Slack messages, support tickets. Semantic search handles the vocabulary mismatch problem: an engineer searching for “memory leak” finds documents about “heap exhaustion” and “GC pressure” because the concepts are semantically related.
Matching incoming support tickets to relevant knowledge base articles or past resolved tickets. A ticket saying “app crashes when I upload large files” semantically matches documentation about “file size limits and upload error handling.”
text-embedding-3-small, encoding 1 million chunks costs approximately $0.02 via the API but takes time to process.