The Metadata-Driven Retrieval Metrics Harness is an evaluator that computes standard information retrieval metrics—NDCG, MRR, Precision, Recall, and MAP—from metadata attached to each evaluation example. Instead of re-running the retrieval pipeline during evaluation, it consumes pre-recorded retrieved_ids and relevant_ids metadata fields, making offline tests mirror the exact retrieval results observed in production traces.
This design separates retrieval evaluation from generation evaluation. The retrieval stage is scored on whether it surfaced the right documents; the generation stage is scored on whether it used those documents well. By recording retrieval metadata at query time and replaying it during evaluation, teams can diagnose whether quality problems originate in retrieval (wrong documents) or generation (wrong use of right documents).
The harness supports configurable thresholds for each metric, enabling teams to set per-metric pass/fail criteria. A retrieval stage might need to maintain NDCG@10 above 0.7 and Recall@20 above 0.85 to pass. When any metric drops below its threshold, the evaluation fails with a detailed breakdown showing which metrics missed and by how much, providing actionable signal for debugging retrieval regressions.
NDCG measures ranking quality by comparing the retrieved order against an ideal ranking. It accounts for both relevance and position: relevant documents appearing earlier in the list score higher than those appearing later. The “discounted” part uses a logarithmic penalty for rank position, and “normalized” means the score is divided by the ideal DCG to produce a value between 0 and 1. NDCG@k evaluates only the top k results.
MRR measures how quickly the system returns the first relevant result. The reciprocal rank for a single query is 1/r where r is the rank of the first relevant document. MRR averages this across all queries. An MRR of 1.0 means the first result is always relevant; 0.5 means the first relevant result is typically at position 2. MRR is particularly useful for navigational queries where users want the single best answer.
Precision@k measures what fraction of the top k retrieved documents are relevant. Recall@k measures what fraction of all relevant documents appear in the top k results. These metrics capture different failure modes: low precision means the system returns too much noise, while low recall means it misses relevant documents. The k parameter is critical—Precision@5 and Precision@100 tell very different stories about system quality.
MAP computes the average precision at each position where a relevant document is found, then averages across all queries. Unlike Precision@k, MAP considers the full ranking and rewards systems that cluster relevant documents at the top. It provides a single-number summary of ranking quality that is sensitive to both precision and recall, making it a popular offline evaluation metric.
Rather than invoking the retrieval pipeline during evaluation, the harness reads retrieval results from metadata fields (retrieved_ids, relevant_ids, optionally relevance_scores) attached to each example. This metadata is recorded during production serving or batch inference, creating a snapshot of the retrieval behavior that can be evaluated repeatedly offline without re-running expensive vector searches.
The harness extracts retrieval metadata from each example, computes per-query metrics, and aggregates them into a pass/fail result with detailed per-metric breakdowns.
package eval
import "math"
func ComputeNDCG(retrievedIDs []string, relevantSet map[string]float64, k int) float64 {
if k <= 0 || len(retrievedIDs) == 0 {
return 0
}
if k > len(retrievedIDs) {
k = len(retrievedIDs)
}
dcg := 0.0
for i := 0; i < k; i++ {
rel, ok := relevantSet[retrievedIDs[i]]
if !ok {
continue
}
dcg += (math.Pow(2, rel) - 1) / math.Log2(float64(i+2))
}
sortedRels := sortDescending(relevantSet)
idcg := 0.0
limit := k
if limit > len(sortedRels) {
limit = len(sortedRels)
}
for i := 0; i < limit; i++ {
idcg += (math.Pow(2, sortedRels[i]) - 1) / math.Log2(float64(i+2))
}
if idcg == 0 {
return 0
}
return dcg / idcg
}
func ComputeMRR(retrievedIDs []string, relevantSet map[string]float64) float64 {
for i, id := range retrievedIDs {
if _, ok := relevantSet[id]; ok {
return 1.0 / float64(i+1)
}
}
return 0
}
func ComputePrecisionAtK(retrievedIDs []string, relevantSet map[string]float64, k int) float64 {
if k <= 0 {
return 0
}
if k > len(retrievedIDs) {
k = len(retrievedIDs)
}
relevant := 0
for i := 0; i < k; i++ {
if _, ok := relevantSet[retrievedIDs[i]]; ok {
relevant++
}
}
return float64(relevant) / float64(k)
}
func ComputeRecallAtK(retrievedIDs []string, relevantSet map[string]float64, k int) float64 {
if len(relevantSet) == 0 {
return 0
}
if k > len(retrievedIDs) {
k = len(retrievedIDs)
}
found := 0
for i := 0; i < k; i++ {
if _, ok := relevantSet[retrievedIDs[i]]; ok {
found++
}
}
return float64(found) / float64(len(relevantSet))
}
func ComputeMAP(retrievedIDs []string, relevantSet map[string]float64) float64 {
if len(relevantSet) == 0 {
return 0
}
sumPrecision := 0.0
relevantSoFar := 0
for i, id := range retrievedIDs {
if _, ok := relevantSet[id]; ok {
relevantSoFar++
sumPrecision += float64(relevantSoFar) / float64(i+1)
}
}
return sumPrecision / float64(len(relevantSet))
}package eval
import (
"context"
"fmt"
"strings"
)
type MetricThreshold struct {
Name string
K int
Threshold float64
}
type RetrievalMetricsEvaluator struct {
thresholds []MetricThreshold
}
func NewRetrievalMetricsEvaluator(thresholds []MetricThreshold) *RetrievalMetricsEvaluator {
return &RetrievalMetricsEvaluator{thresholds: thresholds}
}
func (e *RetrievalMetricsEvaluator) Name() string {
return "retrieval_metrics"
}
func (e *RetrievalMetricsEvaluator) Evaluate(
ctx context.Context, ex Example, out Output,
) (EvalResult, error) {
retrievedIDs, err := extractStringSlice(ex.Input, "retrieved_ids")
if err != nil {
return EvalResult{}, err
}
relevantSet, err := extractRelevantSet(ex.Input)
if err != nil {
return EvalResult{}, err
}
allPassed := true
var failures []string
totalScore := 0.0
for _, t := range e.thresholds {
var score float64
switch t.Name {
case "ndcg":
score = ComputeNDCG(retrievedIDs, relevantSet, t.K)
case "mrr":
score = ComputeMRR(retrievedIDs, relevantSet)
case "precision":
score = ComputePrecisionAtK(retrievedIDs, relevantSet, t.K)
case "recall":
score = ComputeRecallAtK(retrievedIDs, relevantSet, t.K)
case "map":
score = ComputeMAP(retrievedIDs, relevantSet)
}
totalScore += score
if score < t.Threshold {
allPassed = false
failures = append(failures,
fmt.Sprintf("%s@%d=%.3f < %.3f",
t.Name, t.K, score, t.Threshold))
}
}
avgScore := totalScore / float64(len(e.thresholds))
return EvalResult{
Pass: allPassed,
Score: avgScore,
Reason: strings.Join(failures, "; "),
Keywords: failures,
}, nil
}
func extractStringSlice(input map[string]interface{}, key string) ([]string, error) {
raw, ok := input[key].([]interface{})
if !ok {
return nil, fmt.Errorf("missing or invalid %s", key)
}
result := make([]string, len(raw))
for i, v := range raw {
result[i] = fmt.Sprint(v)
}
return result, nil
}
func extractRelevantSet(input map[string]interface{}) (map[string]float64, error) {
rawIDs, ok := input["relevant_ids"].([]interface{})
if !ok {
return nil, fmt.Errorf("missing relevant_ids")
}
relevantSet := make(map[string]float64, len(rawIDs))
for _, id := range rawIDs {
relevantSet[fmt.Sprint(id)] = 1.0
}
if scores, ok := input["relevance_scores"].(map[string]interface{}); ok {
for id, score := range scores {
if s, ok := score.(float64); ok {
relevantSet[id] = s
}
}
}
return relevantSet, nil
}In a typical AI platform project, the retrieval metrics harness runs as a separate evaluation pass from generation quality evaluators. Every production query logs its retrieved_ids and the human-annotated relevant_ids (when available) to a structured trace store. The nightly evaluation pipeline pulls these traces and runs the metrics harness to detect retrieval regressions before they compound into generation quality drops.
The harness is also used for A/B testing retrieval configurations. When the team experiments with a new embedding model or re-ranking strategy, they run both configurations on the same query set and compare the metrics side-by-side. The thresholding mechanism makes this actionable: if the new configuration’s NDCG@10 drops below 0.7 on any query segment, the experiment is automatically flagged for review.
For production monitoring, a lightweight version of the harness runs in real-time on sampled queries. It computes MRR and Precision@5 on queries where relevance labels are available (from click-through data or explicit feedback), feeding these into the observability dashboard. Sustained drops trigger alerts that prompt the retrieval team to investigate.
The harness supports both binary relevance (a document is relevant or not) and graded relevance (relevance scores from 0 to 3). NDCG benefits significantly from graded relevance because it can distinguish between “somewhat relevant” and “highly relevant” documents. When graded labels are available, store them in the relevance_scores metadata field; the harness will use them for NDCG and fall back to binary for Precision/Recall.
Different query types have different retrieval difficulty. Factual queries (“what is X?”) are easy to retrieve for; multi-hop reasoning queries are harder. The harness can be instantiated with different threshold configurations per segment, allowing tighter standards for easy queries and looser ones for known-hard segments.
The metadata-driven approach ensures offline evaluation matches online behavior exactly. The retrieved_ids field contains the actual documents returned by the production retrieval pipeline, not a re-execution of the search. This eliminates a common evaluation pitfall where offline tests use a different index snapshot than production, causing metric discrepancies.