Retrieval-Augmented Generation adds significant latency compared to a direct LLM call. Every RAG query must embed the question, search a vector database, optionally rerank results, construct the augmented prompt, and then wait for the LLM to generate a response over a longer input. Research shows that RAG-specific components — retrieval, reranking, and prompt construction — can account for 25% to 45% of the total end-to-end response time, and in aggressive re-retrieval configurations, up to 97%.
Understanding where time is spent in a RAG pipeline is critical for engineering teams. A naive implementation might deliver 5–10 second response times, which is unacceptable for interactive applications. But blindly optimizing the wrong stage wastes engineering effort — cutting embedding time by 50% matters little when LLM generation dominates 70%+ of total latency. Conversely, for voice-based or streaming applications where time-to-first-token (TTFT) matters most, retrieval latency becomes the bottleneck since it entirely blocks generation from starting.
This topic breaks down where latency comes from in RAG systems, the fundamental tradeoffs between speed and quality at each stage, and the optimization strategies that production systems use to bring response times into acceptable ranges.
A typical RAG pipeline has four sequential latency-contributing stages. Here’s a representative breakdown from production benchmarks:
Embedding: 150–300ms (8–15%) — Converting the query to a vector.
Retrieval: 100–400ms (5–20%) — Vector similarity search + metadata filtering.
Reranking: 100–500ms (5–25%) — Cross-encoder rescoring of top-k candidates.
Generation: 1000–3000ms (50–75%) — LLM processing the augmented prompt and generating tokens.
Total: 1500–4200ms end-to-end.
The critical insight: generation dominates total latency because the LLM must process both the retrieved chunks (often 2000–5000 tokens of context) plus generate the response. More retrieved context means more input tokens, which means higher prefill latency before the first token appears.
Every retrieval decision involves trading accuracy for speed:
Reranking is the single highest-impact quality improvement available, but it comes with real cost:
Cross-encoder reranking of 50 candidates adds 50–100ms of latency but improves precision by 10–30% compared to vector search alone. In some configurations, reranking accounts for 28–87% of the retrieval stage’s total time. The tradeoff: you can rerank fewer candidates (faster, less accurate) or more candidates (slower, more accurate).
More context generally improves answer quality but directly increases generation latency:
For streaming applications, TTFT is often more important than total generation time. Users perceive a system as fast when the first token arrives quickly, even if the complete response takes several seconds. RAG uniquely penalizes TTFT because all retrieval steps must complete before generation can begin:
Timeline (non-streaming direct LLM call):
|-- Generation (TTFT: ~200ms) ---------|
Timeline (RAG with streaming):
|-- Embed (200ms) --|-- Search (150ms) --|-- Rerank (100ms) --|-- Generation (TTFT: ~650ms) ---|
^
First token delayed by 450ms of retrievalCaching is the single most effective latency reduction technique for RAG systems. Three levels of caching apply:
Several pipeline stages can overlap or run in parallel:
# Sequential (slow):
embedding = await embed(query) # 200ms
results = await vector_search(embedding) # 150ms
reranked = await rerank(query, results) # 100ms
response = await generate(query, reranked)# 2000ms
# Total: 2450ms
# Parallel hybrid search:
embedding = await embed(query) # 200ms
vector_results, bm25_results = await asyncio.gather(
vector_search(embedding), # 150ms (parallel)
bm25_search(query) # 120ms (parallel)
) # 200ms (max of both)
fused = rrf_fusion(vector_results, bm25_results) # 5ms
response = await generate(query, fused) # 2000ms
# Total: 2405ms (saved ~70ms on search)Less context means faster generation. Strategies include:
The model itself is the biggest latency lever: