Two-tower retrieval is the production pattern behind many large recommendation and semantic-search systems. One encoder turns the request side into a vector: a search phrase, a user profile, a session, or a free-form description of intent. A second encoder turns candidate items into vectors: documents, products, videos, books, ads, or other catalog entries. At serving time the system compares the request vector against a prebuilt item-vector index and returns the nearest candidates.
The Viberary project is a compact example of this architecture. It uses a Sentence Transformers asymmetric MS MARCO model to encode natural-language prompts and a book corpus into the same retrieval space, stores book embeddings in Redis Search, and uses HNSW nearest-neighbor search to return book recommendations based on descriptive language rather than genre or exact title matching. The important transferable pattern is not the book domain; it is the decoupled query tower, item tower, offline embedding job, ANN index, and online retrieval path.
This page focuses on two-tower retrieval as a candidate-generation layer. It is different from a cross-encoder reranker, which evaluates each query-item pair jointly and is usually too expensive to run against millions of items. The two-tower model sacrifices some interaction detail so item embeddings can be computed ahead of time, indexed, cached, and searched in milliseconds.
A two-tower system has separate computation paths for the request and item sides. In a search application those towers may encode a query and a document. In a recommender they may encode a user/session and an item. The towers may share weights, use different parameters, or use a pretrained asymmetric retriever where one prompt style is intended for queries and another is intended for passages or catalog records.
The towers do not jointly attend over both inputs. They interact only after each side has been converted into a vector, usually through dot product, cosine similarity, or another vector-distance metric. This late-interaction design is what makes the system scalable: item vectors can be produced offline and stored in an ANN index while only the request vector must be computed live.
Two-tower retrieval is usually the high-recall stage in a cascade. It narrows a huge catalog to hundreds or thousands of plausible candidates. A later ranker can then apply business rules, freshness, personalization features, diversity, safety filters, or a cross-encoder reranker to produce the final order.
The model learns a vector space where matching request-item pairs are close. The serving system turns that learned space into an indexable artifact: item IDs, vectors, metadata, and an ANN structure. Production quality depends on both halves.
The item tower runs over every catalog item and writes a stable vector for each item ID. Viberary generates corpus embeddings for book records and loads them into Redis Search, but the same shape applies to product catalogs, media libraries, support articles, or enterprise documents. The offline job should be versioned because a model change invalidates the vector space and requires a coordinated re-embedding and index rebuild.
from sentence_transformers import SentenceTransformer
model = SentenceTransformer("msmarco-distilbert-base-v3")
def item_text(book):
shelves = ", ".join(book["popular_shelves"][:20])
return f"{book['title']} {book.get('description', '')} {shelves}"
vectors = []
for book in books:
vector = model.encode(item_text(book), normalize_embeddings=True)
vectors.append({
"id": book["book_id"],
"title": book["title"],
"embedding": vector.tolist(),
})
index.upsert(vectors)At query time the request tower encodes the user phrase, session, or profile into the same vector space. The system then searches the ANN index for nearest item vectors. For natural-language recommendation, this lets users describe intent with attributes such as mood, style, use case, or constraints instead of matching controlled taxonomy values.
def recommend(prompt, k=20):
query_vector = model.encode(prompt, normalize_embeddings=True)
candidates = index.knn_search(
vector=query_vector,
k=k,
metric="cosine",
filters={"available": True},
)
return candidatesThe first retrieval pass returns candidates, not necessarily the final order. A production system often reranks those results with additional features: exact title matches, popularity, recency, price, availability, user history, content safety, or a cross-encoder score on the top few dozen results. This keeps the two-tower index fast while still allowing the final ranking layer to reason about details that the vector space may blur.
Two-tower retrieval is a strong fit when users know the feel of what they want but not the exact taxonomy. A book, movie, product, or documentation recommender can accept descriptive phrases and retrieve candidates whose learned embeddings align with those descriptions. This avoids forcing users into brittle category filters before the system understands intent.
When the catalog has millions of items, the system cannot run an expensive ranker against every candidate. Two-tower retrieval gives the serving path a fast first pass: compute one request vector, search an ANN index, and hand a bounded candidate set to later stages. This is why the architecture appears in search engines, ad retrieval, feed pre-ranking, and e-commerce recommendation systems.
The item tower can be built from domain-specific fields rather than plain document text. For books, shelf labels and descriptions are useful. For products, titles, attributes, reviews, and taxonomy values may be combined. For internal knowledge search, document titles, summaries, entities, and access metadata can be embedded or filtered. The model sees a retrieval representation tailored to the domain.
Vectors from different models, pooling strategies, normalization settings, or input templates are not safely comparable. Store a model version with every index and reject mixed-version writes unless you are intentionally running a migration. A blue-green index swap is usually safer than mutating the live index in place.
type VectorRecord struct {
ID string
ModelVersion string
Vector []float32
Metadata map[string]string
}
func Upsert(record VectorRecord, activeVersion string) error {
if record.ModelVersion != activeVersion {
return fmt.Errorf("vector version %s does not match active index %s",
record.ModelVersion, activeVersion)
}
return vectorIndex.Upsert(record.ID, record.Vector, record.Metadata)
}The retriever should be tuned for recall and latency; the ranker should be tuned for final utility. Mixing these responsibilities leads to fragile systems. For example, pushing business penalties into the vector text can distort retrieval quality, while ignoring those penalties until after ranking can surface unavailable or unsafe candidates. Keep hard filters explicit and apply final ordering in a separate stage.
The item tower only sees the text or features you feed it. If the template overweights titles, recommendations become title-centric. If it overweights noisy tags, the vector space becomes unstable. Build templates that preserve the fields users actually search by, then evaluate retrieval on realistic queries before scaling the index.