Mean Reciprocal Rank (MRR) is one of the most widely used metrics for evaluating ranked retrieval systems. It measures how quickly a system surfaces the first relevant result for a given query. In the context of AI engineering, MRR is critical for evaluating search engines, question-answering systems, and the retrieval components of RAG (Retrieval-Augmented Generation) pipelines.
The core intuition behind MRR is straightforward: if the first relevant document appears at rank 1, the system performed perfectly for that query. If it appears at rank 5, the system did okay but not great. If no relevant document appears in the top results, the system failed entirely. MRR captures this intuition mathematically by taking the reciprocal of the rank position of the first relevant result and averaging it across all queries.
MRR is particularly valuable in scenarios where users care primarily about a single correct answer rather than browsing through multiple results. Think of voice assistants, FAQ bots, or autocomplete systems—users want the right answer immediately, not a list to browse through. This makes MRR the go-to metric for evaluating these kinds of systems.
Before understanding MRR, you need to understand its building block: the Reciprocal Rank. For a single query, the reciprocal rank is the multiplicative inverse of the rank position where the first relevant document appears:
RR(q) = 1 / rank_i where rank_i is the position of the first relevant document in the ranked result list for query q.
If the first relevant result is at position 1, RR = 1/1 = 1.0. If it’s at position 2, RR = 1/2 = 0.5. At position 3, RR = 1/3 ≈ 0.333. If no relevant result exists in the list, RR = 0.
MRR is simply the average of reciprocal ranks across all queries in an evaluation set. Given a set of queries Q:
MRR = (1 / |Q|) * Σ (1 / rank_i) for all queries i in Q, where |Q| is the total number of queries.
Consider evaluating a search system across three queries:
Query A: Results = [irrelevant, irrelevant, RELEVANT, irrelevant, irrelevant]
First relevant at position 3 → RR = 1/3 = 0.333
Query B: Results = [RELEVANT, irrelevant, irrelevant, irrelevant, irrelevant]
First relevant at position 1 → RR = 1/1 = 1.000
Query C: Results = [irrelevant, irrelevant, irrelevant, irrelevant, RELEVANT]
First relevant at position 5 → RR = 1/5 = 0.200
MRR = (0.333 + 1.000 + 0.200) / 3 = 0.511An MRR of 0.511 tells us that on average the first relevant result appears somewhere between position 1 and 2. The closer MRR is to 1.0, the better the system is at placing relevant results at the very top of the ranked list.
In practice, you often evaluate MRR with a cutoff parameter K (written as MRR@K). This means you only look at the top K results. If no relevant document appears in the top K positions, the reciprocal rank for that query is 0. Common values are MRR@5, MRR@10, and MRR@20.
Here’s a practical implementation you might use in a RAG evaluation pipeline:
def compute_mrr(queries_results: list[list[bool]], k: int = 10) -> float:
"""
Compute MRR@K.
queries_results: list of lists where each inner list contains
boolean relevance judgments (True/False) in rank order.
k: cutoff position
"""
reciprocal_ranks = []
for results in queries_results:
rr = 0.0
for rank, is_relevant in enumerate(results[:k], start=1):
if is_relevant:
rr = 1.0 / rank
break
reciprocal_ranks.append(rr)
return sum(reciprocal_ranks) / len(reciprocal_ranks)
# Example usage
results = [
[False, False, True, False, False], # Query A: relevant at rank 3
[True, False, False, False, False], # Query B: relevant at rank 1
[False, False, False, False, True], # Query C: relevant at rank 5
]
print(f"MRR@10: {compute_mrr(results, k=10):.3f}") # Output: MRR@10: 0.511MRR is one of the most common metrics for evaluating the retriever component of a RAG system. When a user asks a question, the retriever needs to fetch the most relevant chunks from the vector database. MRR measures whether the chunk that actually contains the answer is returned at position 1 (ideal) or buried further down the list. A low MRR in your RAG pipeline is a strong signal that your embedding model, chunking strategy, or similarity search configuration needs tuning.
Web search engines and internal enterprise search systems use MRR as a primary metric. When a user types a query, they overwhelmingly click on the first result—Google reports that the first organic result receives approximately 27.6% of clicks, while the tenth result gets only 2.4%. MRR directly reflects this user behavior pattern by rewarding systems that place the right answer at the top.
For QA systems where there is typically one correct answer, MRR perfectly captures performance. If a user asks “What is the capital of France?” and the system returns “Paris” at position 1, that’s perfect. If “Paris” appears at position 3 after “Lyon” and “Marseille,” MRR penalizes this proportionally.
Autocomplete systems, product recommendations, and entity linking tasks all benefit from MRR evaluation. In these scenarios, the user typically wants a single correct suggestion (the exact product, the correct entity match), making MRR more appropriate than metrics that evaluate the entire ranked list.
1/rank of the first relevant document across all queries.MRR@K with a cutoff to bound evaluation to realistic result set sizes (commonly K=5, 10, or 20).