Embeddings are dense vector representations that capture the semantic meaning of data — text, images, audio, or any structured input — in high-dimensional numerical space. Instead of treating words as opaque symbols (like dictionary index lookups), embeddings map each piece of data to an array of floating-point numbers where geometric proximity reflects semantic similarity. The word “king” and the word “monarch” end up near each other in the vector space, while “king” and “toaster” are far apart.
For AI engineers, embeddings are the bridge between human-readable content and the mathematical operations that machine learning models actually perform. Every modern retrieval-augmented generation (RAG) pipeline, recommendation system, semantic search engine, and classification model depends on high-quality embeddings. Understanding how they are created, how to choose between embedding models, and how to measure similarity between vectors is foundational to building any AI-powered system.
The fundamental insight behind embeddings is the distributional hypothesis: words that appear in similar contexts tend to have similar meanings. Neural networks exploit this by learning to predict surrounding words (or masked tokens) from massive corpora, and the internal representations the network develops during training become the embeddings themselves.
An embedding model takes an input — a word, sentence, paragraph, or entire document — and outputs a fixed-length vector of floating-point numbers. For example, OpenAI’s text-embedding-3-small model produces 1,536-dimensional vectors, while text-embedding-3-large produces 3,072 dimensions. Each dimension captures some learned aspect of meaning, though individual dimensions are not human-interpretable.
An embedding is a mapping f: input → R^d where d is the dimensionality of the embedding space. Semantically similar inputs produce vectors that are geometrically close together, while dissimilar inputs produce vectors that are far apart.
Modern embedding models are typically trained using contrastive learning. The model is shown pairs of inputs that are semantically related (positive pairs) and pairs that are unrelated (negative pairs). During training, the model learns to push positive pairs closer together in vector space and negative pairs further apart. This process is repeated across millions or billions of examples until the model develops robust semantic representations.
Earlier approaches like Word2Vec (2013) and GloVe (2014) produced word-level embeddings by predicting surrounding words in a context window. Modern transformer-based models like those from OpenAI, Cohere, and Google produce contextual embeddings — the same word gets different vectors depending on its surrounding context. The word “bank” in “river bank” and “bank account” receives two distinct embeddings.
Once data is embedded, you need a way to compare vectors. The three most common distance metrics are:
import numpy as np
# Two embedding vectors (simplified to 5 dimensions)
vec_a = np.array([0.12, 0.85, -0.34, 0.67, 0.21])
vec_b = np.array([0.15, 0.80, -0.30, 0.70, 0.18])
# Cosine similarity
cos_sim = np.dot(vec_a, vec_b) / (np.linalg.norm(vec_a) * np.linalg.norm(vec_b))
print(f"Cosine similarity: {cos_sim:.4f}") # ~0.9985 (very similar)
# Euclidean distance
euclidean = np.linalg.norm(vec_a - vec_b)
print(f"Euclidean distance: {euclidean:.4f}") # ~0.0860 (very close)Embedding dimensionality is a critical design choice. Higher dimensions can capture more nuanced semantic relationships but increase storage, computation, and indexing costs. OpenAI’s third-generation models introduced Matryoshka Representation Learning, allowing you to truncate a 3,072-dimensional vector to 256 dimensions with only modest accuracy loss. Remarkably, text-embedding-3-large shortened to 256 dimensions still outperforms the older text-embedding-ada-002 at its full 1,536 dimensions on the MTEB benchmark.
Start with lower-dimensional embeddings (e.g., 256 or 512 dimensions) for prototyping. Only scale up if retrieval quality metrics demand it. Halving dimensions roughly halves your vector storage costs and doubles search speed.
The embedding model landscape has expanded significantly:
text-embedding-3-small: 1,536 dimensions, 62.3% on MTEB, $0.02 per million tokens. Best balance of cost and quality.text-embedding-3-large: 3,072 dimensions, 64.6% on MTEB. Supports flexible dimensionality via the dimensions API parameter.embed-v3: Strong multilingual support with separate query/document embedding modes for asymmetric retrieval.text-embedding-004: 768 dimensions, competitive on multilingual benchmarks, integrated with Vertex AI.BAAI/bge-large-en-v1.5): Self-hostable models that avoid API costs and data-privacy concerns.Traditional keyword search fails when users phrase queries differently than the source material. Embeddings solve this by comparing meaning rather than exact words. A query for “how to fix a broken pipe” will retrieve documents about “plumbing repair” even if those exact words don’t appear in the query. The query and all documents are embedded into the same vector space, and the closest vectors are returned as results.
In RAG pipelines, embeddings are the retrieval engine. Documents are chunked, embedded, and stored in a vector database. When a user asks a question, the query is embedded and the nearest document chunks are retrieved, then injected into the LLM’s context window as grounding material. The quality of the embeddings directly determines the quality of the retrieved context — and therefore the quality of the generated answer.
Embeddings enable unsupervised clustering of documents by topic. Running K-means or DBSCAN on a corpus of embedded documents reveals natural topic groupings without manual labeling. For classification, embedded text can be fed into a simple logistic regression or SVM classifier, often achieving accuracy comparable to fine-tuned models at a fraction of the cost.
Products, articles, or content items are embedded alongside user preference profiles. Recommendations are generated by finding items whose embeddings are closest to the user’s embedding, capturing nuanced taste similarity that collaborative filtering alone cannot.
In support ticket systems or monitoring pipelines, embeddings of incoming messages can be compared against historical norms. Messages with embeddings far from any cluster center may indicate novel issues requiring immediate attention.