Precision and recall are the two foundational metrics for evaluating any information retrieval system — from traditional search engines to the retrieval stages of RAG pipelines. They answer two fundamentally different questions: “Of the documents I retrieved, how many were actually relevant?” (precision) and “Of all the relevant documents that exist, how many did I manage to retrieve?” (recall).
These metrics are essential for AI engineers because the quality of your retrieval stage directly determines the quality of your LLM’s generated output. A retrieval system that returns irrelevant chunks wastes precious context window tokens and may cause the model to hallucinate or provide incorrect answers. A system that misses critical relevant documents produces incomplete or inaccurate responses. Understanding and measuring this tradeoff is the foundation of building effective search and RAG systems.
In the context of modern AI systems, precision and recall take on special importance because they are computed at specific cutoff points (Precision@k and Recall@k), reflecting the practical reality that you’re only passing a fixed number of retrieved documents to the LLM.
Every document in your corpus falls into one of four categories relative to a given query:
Precision = TP / (TP + FP)
“Of everything I retrieved, what fraction was actually relevant?” A precision of 0.80 means 80% of your retrieved documents were relevant — only 20% were noise.
Recall = TP / (TP + FN)
“Of all relevant documents that exist, what fraction did I actually retrieve?” A recall of 0.60 means you found 60% of all relevant documents — 40% were missed.
Imagine you have a corpus of 1000 documents about various programming topics. A user queries “Python async/await patterns”. Suppose 15 documents in the corpus are genuinely relevant to this query.
Your retrieval system returns 10 documents. Upon inspection:
This gives us:
In practice, retrieval systems return ranked lists and we evaluate at specific cutoff points. Precision@k and Recall@k measure these metrics considering only the top-k retrieved results:
Precision@k = (relevant documents in top k) / k
Recall@k = (relevant documents in top k) / (total relevant documents)
For a RAG pipeline that passes 5 chunks to the LLM, Precision@5 and Recall@5 directly measure how well your retrieval stage serves the generator.
Consider this ranked result for our Python async query (relevant marked with ✓):
Rank 1: ✓ "Python asyncio Tutorial"
Rank 2: ✓ "Understanding async/await in Python 3"
Rank 3: ✗ "Python Package Management Guide"
Rank 4: ✓ "Concurrency Patterns with asyncio"
Rank 5: ✗ "JavaScript async/await Explained"
Rank 6: ✓ "Python Event Loop Deep Dive"
Rank 7: ✗ "Python Type Hints Overview"
Rank 8: ✓ "Async Database Queries in Python"
Rank 9: ✓ "Python Threading vs asyncio"
Rank 10: ✗ "General Python Best Practices"
Precision@5 = 3/5 = 0.60
Precision@10 = 6/10 = 0.60
Recall@5 = 3/15 = 0.20
Recall@10 = 6/15 = 0.40Precision and recall are fundamentally in tension. Here’s why:
In a RAG pipeline, this tradeoff has direct consequences: retrieving too few chunks (high precision, low recall) means the LLM lacks information to answer correctly. Retrieving too many chunks (high recall, low precision) wastes context window tokens on noise and may confuse the model.
The F1 score provides a single metric that balances precision and recall using their harmonic mean:
F1 = 2 * (Precision * Recall) / (Precision + Recall)
The harmonic mean penalizes extreme imbalances. A system with Precision=0.95 and Recall=0.10 gets F1=0.18, correctly reflecting that one metric being very low makes the system impractical overall. F1 ranges from 0 (worst) to 1 (perfect).
def precision_at_k(retrieved_ids, relevant_ids, k):
"""Precision@k: fraction of top-k results that are relevant."""
top_k = retrieved_ids[:k]
relevant_in_top_k = len(set(top_k) & set(relevant_ids))
return relevant_in_top_k / k
def recall_at_k(retrieved_ids, relevant_ids, k):
"""Recall@k: fraction of all relevant docs found in top-k."""
top_k = retrieved_ids[:k]
relevant_in_top_k = len(set(top_k) & set(relevant_ids))
return relevant_in_top_k / len(relevant_ids) if relevant_ids else 0
def f1_at_k(retrieved_ids, relevant_ids, k):
"""F1@k: harmonic mean of Precision@k and Recall@k."""
p = precision_at_k(retrieved_ids, relevant_ids, k)
r = recall_at_k(retrieved_ids, relevant_ids, k)
return 2 * p * r / (p + r) if (p + r) > 0 else 0
# Example
retrieved = ["doc_1", "doc_5", "doc_3", "doc_8", "doc_2", "doc_7", "doc_4", "doc_10", "doc_6", "doc_9"]
relevant = ["doc_1", "doc_2", "doc_3", "doc_6", "doc_11", "doc_15"]
for k in [3, 5, 10]:
p = precision_at_k(retrieved, relevant, k)
r = recall_at_k(retrieved, relevant, k)
f = f1_at_k(retrieved, relevant, k)
print(f"@{k}: Precision={p:.2f}, Recall={r:.2f}, F1={f:.2f}")
# Output:
# @3: Precision=0.67, Recall=0.33, F1=0.44
# @5: Precision=0.60, Recall=0.50, F1=0.55
# @10: Precision=0.40, Recall=0.67, F1=0.50In a RAG system, you typically pass the top 3-10 retrieved chunks to the LLM. Precision@k at your chosen k value directly predicts how much of the context window is “signal” vs. “noise.” Recall@k tells you whether you’re missing critical information. A common evaluation workflow:
Precision and recall metrics directly inform architectural decisions:
Different applications prioritize different metrics:
The RAGAS framework (Retrieval-Augmented Generation Assessment) uses precision and recall as core components of its evaluation suite, adapted specifically for RAG:
These RAG-specific adaptations build on the fundamental precision/recall concepts covered here, which you’ll explore in detail in later topics on RAGAS metrics.