Search and RAG systems depend on relevance labels: examples that say whether a document, passage, ticket, message, or record satisfies a specific query. These labels train ranking models, calibrate rerankers, and measure retrieval quality. The hard part is scale. Human judges are expensive, inconsistent, slow, and often unable to inspect sensitive enterprise content directly.
Human-calibrated LLM relevance labeling uses a small, carefully reviewed human dataset to tune and validate an LLM judge, then uses that judge offline to produce many more query-document labels. The LLM does not replace the production ranker at query time. Instead, it acts as a teacher that expands training data for faster rankers, such as feature-based learning-to-rank models, dense retrievers, or cross-encoder rerankers.
The Dropbox Dash relevance-labeling system is a concrete production example of this pattern. It starts with human labels on limited internal data, tunes the LLM evaluator against those labels, measures disagreement with metrics such as mean squared error on a graded relevance scale, samples difficult cases for review, and gives the judge tools to resolve missing query context before scoring. The transferable lesson is that LLM labels become useful when they are calibrated, audited, and continuously compared against human judgment.
A relevance label belongs to a pair, not to a document alone. The same document can be highly relevant for one query and useless for another. Production label schemas usually use a graded scale, such as 1 to 5 or 0 to 2, because partially relevant results still matter for ranking metrics and model training.
The calibration set is a small, high-quality group of human-labeled query-document examples. It should be cleaner than the large dataset the LLM will later label. Its job is not to train the final search ranker directly; its job is to expose whether the LLM judge understands the rating scale, domain terminology, edge cases, and product definition of relevance.
Prompting an LLM to score every candidate result at query time is usually too slow and too expensive, especially when the corpus contains millions or billions of documents. A better architecture uses the LLM offline to generate labels, then trains a cheaper ranker that can run in the serving path.
The human reviewers define and audit the judgment function. The LLM scales that function across more examples than humans could label directly. If the human baseline disappears, the system loses its anchor and can drift toward labels that are merely self-consistent rather than correct.
Start by sampling real or representative queries, retrieving candidate documents from several retrieval methods, and asking trained reviewers to assign graded relevance labels. Sampling from multiple rankers matters because a single production ranker will only surface examples it already prefers, leaving blind spots that make future evaluation unfair.
type RelevanceExample = {
query: string;
document_id: string;
document_excerpt: string;
human_grade: 1 | 2 | 3 | 4 | 5;
reviewer_notes?: string;
source_ranker: "bm25" | "dense" | "hybrid" | "reranker";
};
const calibrationSet: RelevanceExample[] = sampleForReview({
queries: productionQueries,
candidateSources: ["bm25", "dense", "hybrid"],
maxPairsPerQuery: 20
});The LLM judge receives the query, candidate document, grading rubric, and sometimes extra context. Its labels are compared against the human grades. Exact agreement is ideal, but near misses should be penalized less than severe disagreements. Mean squared error is useful for a graded scale because a 1-versus-5 mistake is much worse than a 4-versus-5 mistake.
function mse(humanGrades: number[], llmGrades: number[]) {
let total = 0;
for (let i = 0; i < humanGrades.length; i++) {
const error = humanGrades[i] - llmGrades[i];
total += error * error;
}
return total / humanGrades.length;
}
async function scorePair(query: string, doc: string) {
return llm.generateObject({
schema: { grade: "integer_1_to_5", reason: "string" },
prompt: buildRelevanceRubric(query, doc)
});
}Once the judge meets quality thresholds, run it over hundreds of thousands or millions of sampled pairs. The resulting labels can train a learning-to-rank model, fine-tune a reranker, evaluate retrieval variants, or build a durable relevance test collection. Keep a separate human-reviewed test set so the same synthetic labels are not used both to train and to prove quality.
Enterprise assistants need to find the few records that should ground an answer from a very large permissioned corpus. LLM-generated relevance labels let teams train rankers on domain-specific examples without asking humans to label every internal document pair manually.
A calibrated label collection turns ranking changes into measurable experiments. Teams can compare BM25, dense retrieval, hybrid fusion, reranking prompts, filters, chunking policies, and embedding models against the same judged query set using metrics such as NDCG, precision at k, recall at k, and judged-at-k coverage.
When the LLM judge disagrees with humans, the failure cases become prompt-optimization data. Frameworks such as DSPy can optimize instructions or demonstrations against the human-labeled examples, but the same principle also works with manual rubric refinement and targeted few-shot examples.
Do not spend human review uniformly. Prioritize examples where the LLM label conflicts with user behavior, click signals, prior labels, or high-confidence ranker expectations. These disagreements are where reviewer time produces the strongest calibration signal.
function shouldSendToReview(example: LabeledPair) {
const severeJudgeDisagreement =
Math.abs(example.llm_grade - example.human_or_prior_grade) >= 2;
const behaviorConflict =
example.clicked && example.llm_grade <= 2 ||
!example.clicked && example.llm_grade >= 4;
return severeJudgeDisagreement || behaviorConflict;
}Many relevance failures are context failures. Acronyms, product code names, internal tool names, and team-specific terminology can make a query ambiguous. Give the judge a controlled way to retrieve additional context before assigning a label, but log every context lookup so labels remain reproducible.
Every generated label should carry the judge model, prompt version, rubric version, context sources, timestamp, and calibration-set version. When labels are later used to train a ranker, the model artifact should point back to the exact label-generation run.
{
"query_id": "q_18342",
"document_id": "doc_991",
"grade": 4,
"judge_model": "relevance-judge-2026-02",
"rubric_version": "rag_relevance_v5",
"prompt_version": "dspy_mipro_17",
"calibration_set": "human_seed_2026_02",
"context_lookups": ["term_glossary:diet_sprite"],
"label_run_id": "labels_2026_03_01"
}