Retrieval-Augmented Generation (RAG) systems combine a retriever (search engine) with a generator (LLM) to produce answers grounded in specific documents. But how do you know whether the retriever is finding the right documents, and whether the generator is actually using them? RAG evaluators are automated metrics that measure retrieval quality and citation faithfulness, providing quantitative signals for tuning the entire RAG pipeline.
Retrieval evaluators compute information retrieval metrics—NDCG, MRR, Precision, Recall, MAP—by comparing the retrieved document IDs against a set of known relevant document IDs. These metrics tell you whether the retriever is surfacing the right documents at the right ranks. Citation evaluators parse the generated answer for citation references (like [doc_id X]), verify that each citation points to a document actually provided in the context, and measure coverage: what fraction of the provided context was actually cited.
In the evaluation framework, these evaluators are implemented as modular scoring functions that plug into an evaluation harness. Each evaluator receives metadata fields (retrieved_ids, relevant_ids, the generated answer text) and returns a numeric score. The harness runs evaluators across a test dataset and aggregates scores into a report. This enables systematic A/B testing of retriever changes: swap the embedding model, re-run the evaluation, and compare NDCG scores to determine if retrieval improved.
NDCG measures ranking quality by weighting relevant documents by their position: a relevant document at rank 1 contributes more than one at rank 10. The “normalized” part divides by the ideal DCG (perfect ranking), producing a score between 0 and 1. NDCG@k evaluates only the top k results, which is typical since users rarely look past the first page.
MRR measures where the first relevant result appears: 1/rank_of_first_relevant. If the first relevant document is at rank 3, the reciprocal rank is 1/3. Averaged across queries, MRR tells you how quickly users find a relevant result. MRR is simpler than NDCG and ideal when only the first relevant result matters (e.g., question answering).
Precision = (relevant documents retrieved) / (total documents retrieved). Recall = (relevant documents retrieved) / (total relevant documents). Precision measures noise in results; Recall measures completeness. At a given cutoff k, Precision@k and Recall@k provide a snapshot of the retriever’s accuracy/coverage trade-off at that result depth.
MAP computes the average precision at each position where a relevant document appears, then averages across queries. Unlike Precision@k (which picks a single cutoff), MAP considers the full ranked list and rewards retrievers that front-load relevant documents. It is the single most informative metric for comparing retriever quality across a test set.
Citation evaluators verify that generated answers correctly reference source documents. A citation like [doc_id 3] must point to a document that was actually in the retrieval context. The evaluator flags out-of-range IDs (hallucinated citations), uncited context documents (poor coverage), and answers with no citations at all (unsupported claims).
All retrieval metrics require ground-truth relevance judgments: which documents are relevant to each query. These are stored as metadata pairs: retrieved_ids (what the retriever returned) and relevant_ids (what should have been returned). Relevance can be binary (relevant/not relevant) or graded (0–3 scale). Graded relevance enables NDCG to distinguish “somewhat relevant” from “highly relevant.”
A single query’s metrics can be noisy. Reliable evaluation requires hundreds or thousands of query-judgment pairs. The evaluation harness processes them in bulk, computing per-query metrics and aggregate statistics (mean, median, percentiles). Statistical significance tests (paired t-test or bootstrap) determine whether differences between two retrievers are meaningful or just noise.
Each retrieval metric is a pure function that takes retrieved IDs (ordered by rank) and relevant IDs (ground truth), and returns a numeric score. These functions are stateless and composable—the harness calls them in any order:
// Python: Core retrieval evaluation metrics
from typing import List, Set
def precision_at_k(retrieved: List[str], relevant: Set[str], k: int) -> float:
"""Fraction of top-k retrieved documents that are relevant."""
if k == 0:
return 0.0
top_k = retrieved[:k]
relevant_in_top_k = sum(1 for doc_id in top_k if doc_id in relevant)
return relevant_in_top_k / k
def recall_at_k(retrieved: List[str], relevant: Set[str], k: int) -> float:
"""Fraction of relevant documents found in top-k results."""
if not relevant:
return 0.0
top_k = set(retrieved[:k])
found = len(top_k & relevant)
return found / len(relevant)
def reciprocal_rank(retrieved: List[str], relevant: Set[str]) -> float:
"""Reciprocal of the rank of the first relevant document."""
for i, doc_id in enumerate(retrieved):
if doc_id in relevant:
return 1.0 / (i + 1)
return 0.0
def average_precision(retrieved: List[str], relevant: Set[str]) -> float:
"""Average precision across all positions with relevant docs."""
if not relevant:
return 0.0
hits = 0
sum_precision = 0.0
for i, doc_id in enumerate(retrieved):
if doc_id in relevant:
hits += 1
sum_precision += hits / (i + 1)
return sum_precision / len(relevant)
def ndcg_at_k(
retrieved: List[str],
relevant: dict[str, int], # doc_id -> relevance grade
k: int,
) -> float:
"""Normalized Discounted Cumulative Gain at rank k."""
import math
def dcg(ranked_ids: List[str], grades: dict[str, int], cutoff: int) -> float:
score = 0.0
for i, doc_id in enumerate(ranked_ids[:cutoff]):
rel = grades.get(doc_id, 0)
score += (2**rel - 1) / math.log2(i + 2) # i+2 because rank starts at 1
return score
actual_dcg = dcg(retrieved, relevant, k)
# Ideal DCG: sort by relevance grade descending
ideal_order = sorted(relevant.keys(), key=lambda d: relevant[d], reverse=True)
ideal_dcg = dcg(ideal_order, relevant, k)
if ideal_dcg == 0:
return 0.0
return actual_dcg / ideal_dcgThe citation evaluator parses generated text for citation references, validates them against the provided context, and computes coverage and faithfulness scores:
// Python: Citation parsing and evaluation
import re
from dataclasses import dataclass
@dataclass
class CitationReport:
total_citations: int
valid_citations: int
invalid_citations: int # cite IDs not in context
context_docs: int # total docs in context
cited_docs: int # distinct docs actually cited
coverage: float # cited_docs / context_docs
faithfulness: float # valid_citations / total_citations
uncited_docs: list[str] # context docs never cited
invalid_refs: list[str] # citation IDs that don't exist
def evaluate_citations(
answer: str,
context_doc_ids: list[str],
citation_pattern: str = r"\[doc_id\s+(\w+)\]",
) -> CitationReport:
"""Parse citations from generated answer and evaluate against context."""
# Extract all citation references
matches = re.findall(citation_pattern, answer)
context_set = set(context_doc_ids)
valid = [m for m in matches if m in context_set]
invalid = [m for m in matches if m not in context_set]
cited_set = set(valid)
uncited = [d for d in context_doc_ids if d not in cited_set]
total = len(matches)
return CitationReport(
total_citations=total,
valid_citations=len(valid),
invalid_citations=len(invalid),
context_docs=len(context_doc_ids),
cited_docs=len(cited_set),
coverage=len(cited_set) / len(context_doc_ids) if context_doc_ids else 0.0,
faithfulness=len(valid) / total if total > 0 else 1.0,
uncited_docs=uncited,
invalid_refs=invalid,
)
# Example usage
answer = """Based on the analysis in [doc_id 3], the system uses
consistent hashing for partitioning. This is further explained in
[doc_id 7] which describes virtual nodes. However, [doc_id 99]
suggests an alternative approach."""
report = evaluate_citations(
answer=answer,
context_doc_ids=["1", "3", "5", "7", "10"],
)
# report.faithfulness = 2/3 = 0.667 (doc_id 99 is invalid)
# report.coverage = 2/5 = 0.4 (only docs 3 and 7 were cited)The evaluation harness orchestrates metric computation across a test dataset. It loads query-judgment pairs, runs each through the retriever, computes all metrics, and produces aggregate statistics:
// Python: Evaluation harness for batch metric computation
from dataclasses import dataclass, field
from statistics import mean, median
@dataclass
class QueryResult:
query_id: str
retrieved_ids: list[str]
relevant_ids: set[str]
relevance_grades: dict[str, int] # for NDCG
generated_answer: str = ""
context_doc_ids: list[str] = field(default_factory=list)
@dataclass
class EvalReport:
num_queries: int
mean_ndcg_10: float
mean_mrr: float
mean_precision_10: float
mean_recall_10: float
mean_map: float
mean_citation_faithfulness: float
mean_citation_coverage: float
def evaluate_batch(results: list[QueryResult], k: int = 10) -> EvalReport:
"""Compute aggregate retrieval and citation metrics."""
ndcg_scores = []
mrr_scores = []
p_scores = []
r_scores = []
ap_scores = []
faith_scores = []
cov_scores = []
for r in results:
ndcg_scores.append(ndcg_at_k(r.retrieved_ids, r.relevance_grades, k))
mrr_scores.append(reciprocal_rank(r.retrieved_ids, r.relevant_ids))
p_scores.append(precision_at_k(r.retrieved_ids, r.relevant_ids, k))
r_scores.append(recall_at_k(r.retrieved_ids, r.relevant_ids, k))
ap_scores.append(average_precision(r.retrieved_ids, r.relevant_ids))
if r.generated_answer and r.context_doc_ids:
cr = evaluate_citations(r.generated_answer, r.context_doc_ids)
faith_scores.append(cr.faithfulness)
cov_scores.append(cr.coverage)
return EvalReport(
num_queries=len(results),
mean_ndcg_10=mean(ndcg_scores),
mean_mrr=mean(mrr_scores),
mean_precision_10=mean(p_scores),
mean_recall_10=mean(r_scores),
mean_map=mean(ap_scores),
mean_citation_faithfulness=mean(faith_scores) if faith_scores else 0.0,
mean_citation_coverage=mean(cov_scores) if cov_scores else 0.0,
)The evaluation framework uses these evaluators to compare retriever configurations. When switching from BM25 to hybrid BM25+vector retrieval, the team runs the evaluation harness on a fixed test set of 500 queries with human-labeled relevance judgments. The report shows NDCG@10 improved from 0.72 to 0.81, confirming the hybrid approach surfaces more relevant documents at higher ranks. Without automated evaluation, this comparison would require manual inspection of thousands of search results.
In production RAG systems, citation faithfulness can degrade when the LLM hallucinates document references. The citation evaluator runs on a sample of production answers to track faithfulness over time. A drop in faithfulness triggers an alert, indicating that the LLM prompt or retrieval context format may need adjustment. This is particularly important after model updates, which can change citation behavior.
Choosing between embedding models (e.g., OpenAI text-embedding-3-small vs. text-embedding-3-large vs. open-source alternatives) is a retrieval quality decision. The evaluation harness computes NDCG and MAP for each model on the same test set, enabling data-driven model selection that balances quality against cost and latency.
RAG retrieval quality depends heavily on chunk size: too small and chunks lack context, too large and they dilute relevance. The evaluator enables systematic testing: generate chunk sets at 256, 512, 1024, and 2048 tokens, build an index for each, run the same queries, and compare Recall@10 and NDCG@10. The optimal chunk size varies by document type and query style.
Binary relevance (relevant/not relevant) is simpler to annotate but loses nuance. Graded relevance (0=irrelevant, 1=marginally relevant, 2=relevant, 3=highly relevant) enables NDCG to reward systems that rank highly relevant documents above marginally relevant ones. For RAG evaluation, graded relevance better captures whether the retrieved documents contain enough information to generate a correct answer.
Different LLMs use different citation formats: [1], [doc_id X], (Source: title), [[ref:id]]. The citation evaluator accepts a configurable regex pattern so it works across citation formats. The pattern is specified in the evaluation config and can be updated without changing evaluator code:
// Python: Configurable citation patterns
CITATION_PATTERNS = {
"bracket_number": r"\[(\d+)\]",
"doc_id": r"\[doc_id\s+(\w+)\]",
"source_ref": r"\[\[ref:(\w+)\]\]",
"footnote": r"\[fn:(\d+)\]",
}
def evaluate_with_pattern(
answer: str,
context_ids: list[str],
pattern_name: str = "doc_id",
) -> CitationReport:
pattern = CITATION_PATTERNS[pattern_name]
return evaluate_citations(answer, context_ids, pattern)Aggregate metrics hide individual query failures. The harness stores per-query metric breakdowns so engineers can identify which queries perform worst. Sorting queries by NDCG@10 ascending shows the retriever’s weakest spots—often queries with unusual vocabulary, ambiguous intent, or sparse relevant documents. These worst-case queries become targets for retrieval improvements.
When comparing two retrievers, a small difference in mean NDCG might not be statistically significant. Bootstrap resampling computes confidence intervals: resample the test set with replacement 1000 times, compute the metric difference each time, and report the 95% confidence interval. If the interval excludes zero, the difference is significant.
// Python: Bootstrap confidence interval for metric comparison
import random
def bootstrap_ci(
scores_a: list[float],
scores_b: list[float],
n_bootstrap: int = 1000,
confidence: float = 0.95,
) -> tuple[float, float, float]:
"""Compute bootstrap confidence interval for mean(A) - mean(B)."""
diffs = []
n = len(scores_a)
for _ in range(n_bootstrap):
indices = [random.randint(0, n - 1) for _ in range(n)]
sample_a = [scores_a[i] for i in indices]
sample_b = [scores_b[i] for i in indices]
diffs.append(mean(sample_a) - mean(sample_b))
diffs.sort()
alpha = (1 - confidence) / 2
lo = diffs[int(alpha * n_bootstrap)]
hi = diffs[int((1 - alpha) * n_bootstrap)]
return mean(diffs), lo, hi # point estimate, lower bound, upper boundIf the test set only labels 3 relevant documents per query but the corpus contains 20, Recall@10 will be artificially high (finding 3/3 = 100%) and a retriever that finds the unlabeled relevant documents gets no credit. Use pooled judgments: have the retriever surface candidates, then judge them, to build a more complete relevance set.
If the embedding model was fine-tuned on the same query-document pairs used for evaluation, metrics will be inflated. Always hold out a separate test set that the retriever has never seen during training. Cross-validation on the evaluation set also helps detect overfitting.
Precision@10 treats all 10 positions equally: a relevant document at rank 1 contributes the same as one at rank 10. For RAG, where only the top 3–5 documents are included in the LLM context, use NDCG (which penalizes low-ranked relevant docs) or Precision@3 instead of Precision@10.
If the citation regex doesn’t match the LLM’s actual citation format, the evaluator reports zero citations and 0% faithfulness. Always validate the citation pattern against sample outputs before running batch evaluation. A single mismatched bracket or space in the regex can cause all citations to be missed.
High retrieval metrics (NDCG, MRR) don’t guarantee good generated answers. The LLM might ignore retrieved context, misinterpret it, or hallucinate despite having the right documents. Retrieval evaluators and citation evaluators measure different failure modes. Both are needed: retrieval metrics catch retriever problems, citation metrics catch generator problems.