A reranker is a second-stage model that takes the initial results from a fast retriever (vector search, BM25, or hybrid) and re-scores them with much higher accuracy. The most common architecture for rerankers is the cross-encoder — a transformer model that processes the query and a candidate document together as a single concatenated input, enabling full cross-attention between every token in both texts.
This is fundamentally different from the bi-encoder architecture used in first-stage retrieval, where the query and document are encoded independently into separate vectors and compared via cosine similarity. Because bi-encoders never see the query and document tokens together, they miss nuanced interactions between them. Cross-encoders capture these interactions at the cost of being far more computationally expensive.
In practice, adding a cross-encoder reranking stage to a RAG pipeline typically improves retrieval accuracy by 20-35%, with an additional latency cost of 200-500ms for reranking the top 20-100 candidates. This accuracy-latency tradeoff makes rerankers one of the highest-impact improvements you can make to a retrieval system.
Understanding the architectural difference is key to understanding why rerankers exist:
Encodes query and document independently into fixed-size vectors. Similarity is computed via dot product or cosine similarity. Can pre-compute document embeddings, making retrieval over millions of documents feasible in milliseconds. But the independent encoding means the model cannot capture token-level interactions between the query and document.
Concatenates the query and document as a single input: [CLS] query [SEP] document [SEP]. The transformer’s self-attention layers allow every query token to attend to every document token and vice versa. Outputs a single relevance score (0 to 1). Cannot pre-compute representations — every query-document pair requires a full forward pass.
The standard pattern in production systems combines both architectures:
This two-stage design gives you the best of both worlds: the speed of bi-encoders for searching millions of documents, and the accuracy of cross-encoders for the final ranking of a small candidate set.
When a cross-encoder processes a query-document pair, the following happens:
[CLS] token’s final hidden state is passed through a linear classification head that outputs a relevance score.This full cross-attention is what makes cross-encoders so much more accurate. Consider the query “apple stock price” — a bi-encoder might embed “apple” near both the fruit and the company, but a cross-encoder can use the context of “stock price” to attend specifically to financial content in the document.
from sentence_transformers import CrossEncoder
# Load a pre-trained cross-encoder
model = CrossEncoder("cross-encoder/ms-marco-MiniLM-L-6-v2", max_length=512)
query = "How does photosynthesis convert sunlight to energy?"
# Candidate documents from first-stage retrieval
candidates = [
"Photosynthesis uses chlorophyll to capture light energy and convert CO2 and water into glucose.",
"Solar panels convert sunlight into electricity using photovoltaic cells made of silicon.",
"The process of photosynthesis occurs in the chloroplasts of plant cells in two stages: light-dependent reactions and the Calvin cycle.",
"Plants need water, sunlight, and carbon dioxide to survive and grow.",
]
# Score each query-document pair
scores = model.predict([(query, doc) for doc in candidates])
# Sort by score descending
ranked = sorted(zip(candidates, scores), key=lambda x: x[1], reverse=True)
for doc, score in ranked:
print(f"Score: {score:.4f} | {doc[:80]}...")The ecosystem of pre-trained reranker models has matured significantly:
cross-encoder/ms-marco-MiniLM-L-6-v2 — The workhorse. Only 22M parameters, fast inference (~5ms per pair on GPU), trained on MS MARCO passage ranking. Best starting point for most use cases.cross-encoder/ms-marco-MiniLM-L-12-v2 — 33M parameters. Slightly more accurate than L-6, roughly 2x slower. Good for applications where latency budget allows it.BAAI/bge-reranker-base — 278M parameters. Part of the BGE family from BAAI. Strong multilingual support, significantly more accurate than MiniLM models but 10x slower.BAAI/bge-reranker-large — 560M parameters. Top-tier accuracy for multilingual reranking. Best for offline or batch processing where latency is less critical.The most impactful use of rerankers is in RAG pipelines. Without reranking, the LLM receives whatever the first-stage retriever returns — which may include marginally relevant or off-topic chunks that waste context window space and degrade answer quality. A reranker ensures only the most relevant chunks reach the generator:
# Typical RAG pipeline with reranking
def rag_with_reranking(query, vector_store, reranker, llm, top_k=5, rerank_k=50):
# Stage 1: Fast retrieval (cast wide net)
candidates = vector_store.similarity_search(query, k=rerank_k)
# Stage 2: Rerank (precision filter)
pairs = [(query, doc.page_content) for doc in candidates]
scores = reranker.predict(pairs)
# Take top-k after reranking
ranked_indices = sorted(range(len(scores)), key=lambda i: scores[i], reverse=True)
top_docs = [candidates[i] for i in ranked_indices[:top_k]]
# Stage 3: Generate with only the best context
context = "\n\n".join([doc.page_content for doc in top_docs])
return llm.generate(f"Context:\n{context}\n\nQuestion: {query}")In production, you need to carefully plan your latency budget. A typical breakdown for a RAG request:
The reranking stage typically adds 10-20% to total latency while delivering 20-35% accuracy improvement — an excellent tradeoff for most applications.
Rerankers aren’t always the right choice:
An emerging pattern uses LLMs themselves as rerankers by prompting them to score or rank documents. Systems like RankGPT and RankLLM ask the LLM to sort a list of passages by relevance. This can outperform traditional cross-encoders on some benchmarks but costs significantly more per query and is slower. It’s most viable when using small, specialized LLMs or when maximum accuracy justifies the cost.
cross-encoder/ms-marco-MiniLM-L-6-v2 (22M params, ~5ms/pair) and scale up to BGE or Cohere models only if you need higher accuracy or multilingual support.