Document re-ranking is the critical second stage in modern retrieval pipelines. A first-stage retriever (BM25, dense retrieval, or hybrid) casts a wide net to identify candidate documents, and a re-ranker then refines the ordering by computing fine-grained relevance scores for each query-document pair. The quality of this re-ranking step directly determines end-user experience—it decides which documents surface at the top of search results, which chunks feed into a RAG pipeline, and which passages a question-answering system selects as evidence.
Traditional cross-encoder re-rankers based on BERT or similar encoder models compute a scalar relevance score by jointly attending to the query and document. These models are effective and efficient at inference time, but they lack the capacity for complex reasoning about why a document is relevant—they produce a number, not an explanation. On the other end of the spectrum, LLM-based re-rankers such as RankGPT and GPT-4 can reason about relevance through natural language, producing chain-of-thought justifications and handling nuanced queries. However, these come at enormous cost: GPT-4 re-ranking is orders of magnitude slower and more expensive than a cross-encoder, making it impractical for production workloads processing millions of queries per day.
DeAR (Dual-Stage Document Reranking with Reasoning Agents via LLM Distillation) bridges this gap. Published at EMNLP 2025 Findings, DeAR distills the reasoning capability of large language models into compact student models (3B–8B parameters) through a novel two-stage training approach. Stage 1 uses pointwise knowledge distillation to transfer token-level relevance signals from a frozen 13B LLaMA teacher into the student. Stage 2 attaches a second LoRA adapter and fine-tunes on 20,000 GPT-4o-generated chain-of-thought ranking permutations, enabling the student to produce natural-language justifications alongside its rankings. The result is a model that approaches GPT-4 accuracy on standard benchmarks while running at a fraction of the cost and latency.
There are two fundamental paradigms for neural re-ranking, and understanding them is essential to grasping how DeAR works:
The re-ranker scores each query-document pair independently, producing a scalar relevance score. Documents are then sorted by score. This is how cross-encoders (BERT, MonoT5) operate. Advantages: simple, parallelizable, and produces calibrated scores. Disadvantage: the model never sees relationships between candidate documents, so it cannot reason about comparative relevance.
The re-ranker takes the full list (or a sliding window) of candidate documents at once and directly produces a permutation—an ordering of the documents from most to least relevant. This is how RankGPT and similar LLM-based approaches operate. Advantages: the model can reason about relative relevance and make direct comparisons. Disadvantage: computationally expensive, context-length limited, and harder to train with supervision.
DeAR combines both paradigms sequentially. Stage 1 trains the student in the pointwise paradigm to produce calibrated relevance scores. Stage 2 then layers listwise reasoning on top, enabling the model to generate ranked permutations with natural-language justifications. This dual-stage approach captures the strengths of both paradigms while mitigating their individual weaknesses.
Knowledge distillation is the process of training a smaller “student” model to mimic the behavior of a larger “teacher” model. In standard distillation, the student learns to reproduce the teacher’s output distribution (soft labels) rather than just the hard labels from a training set. The key insight is that the teacher’s soft probability distribution contains richer information than binary labels—it reveals how confident the teacher is and which alternatives it considers plausible.
DeAR uses a frozen 13B LLaMA model as the teacher for Stage 1 distillation. “Frozen” means the teacher’s weights are not updated during training; it serves only as a source of supervision signals. The student (3B or 8B parameters) learns to match the teacher’s token-level probability distributions over relevance labels. This is more nuanced than learning from binary relevant/not-relevant labels because the teacher communicates degrees of relevance through its probability mass allocation.
LoRA (Low-Rank Adaptation) inserts small trainable matrices into the frozen layers of a pre-trained model. Instead of fine-tuning all parameters (which for an 8B model means updating 8 billion weights), LoRA trains only the low-rank decomposition matrices, typically adding less than 1% new parameters. This dramatically reduces GPU memory requirements and training time.
DeAR uses LoRA in both stages but with a critical architectural choice: Stage 1 and Stage 2 use separate LoRA adapters. The Stage 1 adapter specializes in pointwise relevance scoring. The Stage 2 adapter, attached on top of the Stage 1 base, specializes in listwise reasoning and chain-of-thought generation. At inference time, both adapters are loaded simultaneously, and the model can be used in either pointwise or listwise mode depending on the application requirements.
Using a single adapter for both stages would create competing optimization objectives: pointwise scoring wants precise calibration of a scalar output, while listwise reasoning wants fluent chain-of-thought generation. By separating the adapters, each stage can optimize freely without degrading the other. The Stage 2 adapter benefits from the Stage 1 foundation without overwriting it.
The listwise training data for Stage 2 is generated by GPT-4o. For each training query, GPT-4o is given a set of candidate documents and asked to (1) reason about the relevance of each document in natural language, and (2) produce a ranking permutation. This results in 20,000 training examples, each consisting of a query, a set of documents, a chain-of-thought reasoning trace, and the final permutation. The chain-of-thought is critical: it teaches the student not just what the correct ranking is but why, enabling the student to generalize to new domains and query types.
DeAR’s architecture is a two-stage training pipeline where each stage builds on the previous one. The base model is a pre-trained LLaMA variant (3B or 8B parameters). Each stage attaches a LoRA adapter and optimizes a distinct objective.
In Stage 1, the student learns to produce calibrated pointwise relevance scores by mimicking a frozen 13B LLaMA teacher. The input is a query-document pair formatted as a prompt. Both teacher and student process the same input, and the student is trained to match the teacher’s output distribution.
The training objective is a hybrid loss function combining three components:
L = alpha * L_CE + beta * L_RN + gamma * L_KL
The three loss terms are balanced by hyperparameters alpha, beta, and gamma. Cross-entropy provides the ground-truth anchor, RankNet enforces pairwise ordering constraints, and KL divergence transfers the teacher’s distributional knowledge. The combination produces scores that are both accurate (from CE), well-ordered (from RankNet), and calibrated to match the teacher’s confidence (from KL).
The teacher model is kept frozen throughout training. Its outputs are computed once per batch and cached, so the teacher does not add to the backward-pass computational cost. The LoRA adapter attached during this stage is referred to as LoRA-S1. After Stage 1 completes, the student has learned to produce pointwise relevance scores that closely track the 13B teacher’s assessments while running in a 3B or 8B parameter envelope.
Stage 2 takes the Stage 1 student (base model + LoRA-S1) and attaches a second LoRA adapter, LoRA-S2. The Stage 1 adapter weights are frozen during Stage 2 training—only the new LoRA-S2 parameters are updated. This preserves the pointwise scoring capability acquired in Stage 1.
The training data consists of 20,000 chain-of-thought ranking examples generated by GPT-4o. Each example contains:
The student is fine-tuned with a standard next-token prediction objective (autoregressive language modeling loss) on these examples. The model learns to first generate a reasoning trace and then produce the permutation, mimicking GPT-4o’s reasoning process. This is a form of chain-of-thought distillation: the student learns not just the teacher’s answers but its reasoning patterns.
Because the student already has strong relevance understanding from Stage 1 (pointwise distillation from the 13B teacher), Stage 2 does not need to teach relevance from scratch. It only needs to teach the format of reasoning and permutation generation. This is why a relatively small dataset of 20K examples is sufficient—the heavy lifting of relevance understanding was done in Stage 1.
At inference time, the two stages can be used independently or in sequence. The most effective configuration runs both stages as a cascade:
# Pseudocode: DeAR two-stage inference pipeline
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel
# Load base model with both LoRA adapters
base_model = AutoModelForCausalLM.from_pretrained("meta-llama/Llama-3.1-8B")
model = PeftModel.from_pretrained(base_model, "dear-lora-s1")
model.load_adapter("dear-lora-s2", adapter_name="listwise")
tokenizer = AutoTokenizer.from_pretrained("meta-llama/Llama-3.1-8B")
def stage1_pointwise_score(query: str, documents: list[str]) -> list[float]:
"""Stage 1: Score each document independently."""
model.set_active_adapter("default") # LoRA-S1
scores = []
for doc in documents:
prompt = f"Query: {query}\nDocument: {doc}\nRelevance:"
inputs = tokenizer(prompt, return_tensors="pt")
logits = model(**inputs).logits[:, -1, :]
# Extract probability mass on "Yes"/"No" relevance tokens
yes_score = logits[0, tokenizer.encode("Yes")[0]].item()
no_score = logits[0, tokenizer.encode("No")[0]].item()
score = yes_score / (yes_score + no_score)
scores.append(score)
return scores
def stage2_listwise_rerank(
query: str, documents: list[str], top_k: int = 10
) -> tuple[list[int], str]:
"""Stage 2: Generate chain-of-thought reasoning and permutation."""
model.set_active_adapter("listwise") # LoRA-S2
doc_list = "\n".join(
f"[{i+1}] {doc}" for i, doc in enumerate(documents[:top_k])
)
prompt = (
f"Query: {query}\n\nDocuments:\n{doc_list}\n\n"
f"Rank the documents by relevance. Think step by step, "
f"then provide the ranking as a permutation.\n\nReasoning:"
)
inputs = tokenizer(prompt, return_tensors="pt")
outputs = model.generate(**inputs, max_new_tokens=1024, temperature=0.0)
response = tokenizer.decode(outputs[0], skip_special_tokens=True)
# Parse the permutation and reasoning from the response
reasoning, permutation = parse_response(response)
return permutation, reasoning
def dear_rerank(
query: str, documents: list[str], top_k: int = 10
) -> list[dict]:
"""Full DeAR pipeline: pointwise filter, then listwise reasoning."""
# Stage 1: Pointwise scoring of all candidates
scores = stage1_pointwise_score(query, documents)
# Select top-K candidates by pointwise score
ranked_indices = sorted(
range(len(scores)), key=lambda i: scores[i], reverse=True
)
top_docs = [documents[i] for i in ranked_indices[:top_k]]
# Stage 2: Listwise re-ranking with reasoning
permutation, reasoning = stage2_listwise_rerank(query, top_docs, top_k)
# Map permutation back to original document indices
results = []
for rank, perm_idx in enumerate(permutation):
orig_idx = ranked_indices[perm_idx]
results.append({
"rank": rank + 1,
"document": documents[orig_idx],
"pointwise_score": scores[orig_idx],
"reasoning": reasoning,
})
return resultsThe cascade works as follows: Stage 1 pointwise scoring runs in parallel across all candidate documents (since each is scored independently), producing calibrated relevance scores. The top-K candidates by pointwise score are then passed to Stage 2, which performs listwise re-ranking with chain-of-thought reasoning. This two-pass approach is efficient because Stage 1 filters the large candidate set down to a manageable size before the more expensive Stage 2 processes the survivors.
In enterprise search systems (legal discovery, compliance, internal knowledge bases), users often need to understand why a document was ranked highly. Traditional cross-encoders produce only a score, leaving users to guess. DeAR’s Stage 2 generates natural-language justifications: “Document 3 is ranked first because it directly addresses the plaintiff’s claim regarding contract breach in Section 4.2, whereas Document 7 only mentions breach tangentially in a footnote.” This explainability is increasingly required by regulatory frameworks governing AI-assisted decision-making.
Legal search queries are often complex and nuanced, requiring reasoning about statutory language, precedent, and factual similarity. A pointwise cross-encoder might score a superficially similar case highly based on keyword overlap, while missing that the legal reasoning is actually opposed to the query’s intent. DeAR’s listwise reasoning stage can capture these nuances because it was trained on chain-of-thought traces that reason about document content at a deeper level.
In Retrieval-Augmented Generation, the quality of the re-ranking step directly impacts generation quality. Inserting a DeAR re-ranker between the retriever and the generator improves the signal-to-noise ratio of the context window. The typical architecture is: retrieve top-100 with BM25 or dense retrieval, re-rank with DeAR Stage 1 to select top-20, then apply Stage 2 listwise reasoning to produce the final top-5 passages for the generator. This cascading approach ensures the generator receives the most relevant information in the optimal order, at a fraction of the cost of using GPT-4 directly as the re-ranker.
For open-domain question answering, the re-ranker must identify which passages contain the actual answer versus which merely discuss the topic. DeAR achieves 54.29 Top-1 accuracy on Natural Questions without any Wikipedia-specific fine-tuning, demonstrating strong zero-shot transfer. The listwise reasoning capability is especially useful here because it can compare candidate passages and reason about which one most directly answers the question versus which ones provide background context.
Organizations currently using GPT-4 or similar frontier models for re-ranking can substitute a DeAR student model to achieve comparable accuracy at dramatically reduced cost. A 3B parameter model running on a single GPU processes queries orders of magnitude faster than an API call to GPT-4, with no per-token charges. For high-volume search workloads (millions of queries per day), this translates to significant operational savings while maintaining ranking quality.
The most common deployment pattern is a three-stage cascade: first-stage retriever, pointwise re-ranker, and listwise re-ranker. Each stage narrows the candidate set before the next, more expensive stage processes it.
# Production retrieval pipeline with DeAR re-ranking
from pyserini.search.lucene import LuceneSearcher
from dear_reranking import DeARReranker
# Stage 0: First-stage retrieval (BM25 via Pyserini)
searcher = LuceneSearcher.from_prebuilt_index("msmarco-v1-passage")
# Initialize DeAR with both LoRA adapters
reranker = DeARReranker(
base_model="meta-llama/Llama-3.1-8B",
pointwise_adapter="DataScienceUIBK/dear-lora-pointwise",
listwise_adapter="DataScienceUIBK/dear-lora-listwise",
device="cuda:0",
)
def search(query: str, top_k: int = 5) -> list[dict]:
# Retrieve top-100 candidates with BM25
hits = searcher.search(query, k=100)
candidates = [
{"docid": h.docid, "text": h.raw, "bm25_score": h.score}
for h in hits
]
# Stage 1: Pointwise re-ranking (scores all 100 candidates)
pointwise_scores = reranker.pointwise_score(
query, [c["text"] for c in candidates]
)
for i, c in enumerate(candidates):
c["pointwise_score"] = pointwise_scores[i]
# Sort by pointwise score and take top-20 for listwise stage
candidates.sort(key=lambda c: c["pointwise_score"], reverse=True)
shortlist = candidates[:20]
# Stage 2: Listwise re-ranking with reasoning (top-20 only)
permutation, reasoning = reranker.listwise_rerank(
query, [c["text"] for c in shortlist]
)
# Apply the permutation and return top-K
results = [shortlist[i] for i in permutation[:top_k]]
for rank, r in enumerate(results):
r["final_rank"] = rank + 1
r["reasoning"] = reasoning
return resultsDeAR’s dual-adapter architecture requires careful management of LoRA adapters. The pattern below shows how to load and switch between the pointwise and listwise adapters at inference time:
from transformers import AutoModelForCausalLM, BitsAndBytesConfig
from peft import PeftModel
# Quantization config for memory-efficient inference
bnb_config = BitsAndBytesConfig(
load_in_4bit=True,
bnb_4bit_compute_dtype="float16",
bnb_4bit_quant_type="nf4",
)
# Load base model with 4-bit quantization
base_model = AutoModelForCausalLM.from_pretrained(
"meta-llama/Llama-3.1-8B",
quantization_config=bnb_config,
device_map="auto",
)
# Load Stage 1 (pointwise) adapter as default
model = PeftModel.from_pretrained(
base_model,
"DataScienceUIBK/dear-lora-pointwise",
adapter_name="pointwise",
)
# Load Stage 2 (listwise) adapter alongside
model.load_adapter(
"DataScienceUIBK/dear-lora-listwise",
adapter_name="listwise",
)
# Switch between adapters for each stage
model.set_active_adapter("pointwise") # For Stage 1
# ... run pointwise scoring ...
model.set_active_adapter("listwise") # For Stage 2
# ... run listwise reasoning ...Stage 1 pointwise scoring is embarrassingly parallel—each query-document pair is independent. For production throughput, batch multiple pairs together:
import torch
def batch_pointwise_score(
model, tokenizer, query: str, documents: list[str], batch_size: int = 16
) -> list[float]:
"""Score documents in batches for throughput."""
model.set_active_adapter("pointwise")
all_scores = []
for i in range(0, len(documents), batch_size):
batch_docs = documents[i:i + batch_size]
prompts = [
f"Query: {query}\nDocument: {doc}\nRelevance:"
for doc in batch_docs
]
inputs = tokenizer(
prompts, return_tensors="pt",
padding=True, truncation=True, max_length=512
).to(model.device)
with torch.no_grad():
logits = model(**inputs).logits[:, -1, :]
yes_id = tokenizer.encode("Yes", add_special_tokens=False)[0]
no_id = tokenizer.encode("No", add_special_tokens=False)[0]
yes_logits = logits[:, yes_id]
no_logits = logits[:, no_id]
probs = torch.softmax(
torch.stack([no_logits, yes_logits], dim=1), dim=1
)
scores = probs[:, 1].cpu().tolist()
all_scores.extend(scores)
return all_scoresThe official DeAR implementation is available at github.com/DataScienceUIBK/DeAR-Reranking. The repository includes training scripts for both stages, pre-trained LoRA adapters, evaluation scripts for TREC and BEIR benchmarks, and the GPT-4o-generated chain-of-thought training data. The codebase builds on Hugging Face Transformers and PEFT, making it straightforward to adapt to custom domains.
DeAR demonstrates strong performance across multiple evaluation settings, consistently outperforming open-source baselines and competing with proprietary models at a fraction of the cost.
DeAR achieves a +5.1 nDCG@5 improvement over open-source baselines on the TREC Deep Learning 2020 passage ranking task. This benchmark uses queries from the MS MARCO dataset with deep relevance judgments from NIST assessors, making it one of the most authoritative evaluations of passage ranking quality.
On NovelEval, a benchmark designed to test generalization to novel queries and documents unseen during training, DeAR achieves 90.97 nDCG@10, beating GPT-4 by +3.09 points. This is particularly notable because DeAR’s student model (8B parameters) outperforms GPT-4 (estimated >1T parameters) on a generalization benchmark, demonstrating that the distillation process transfers not just memorized patterns but genuine ranking capability.
On the Natural Questions open-domain QA benchmark, DeAR achieves 54.29 Top-1 accuracy without any Wikipedia-specific fine-tuning. This zero-shot transfer result demonstrates that the distilled model generalizes beyond its training distribution, applying its learned ranking strategies to a completely different domain and task format.
DeAR was evaluated across 8 datasets from the BEIR benchmark, which spans diverse domains including scientific literature (SCIDOCS), question answering (Natural Questions, HotpotQA), fact verification (FEVER, Climate-FEVER), citation prediction (SCIFACT), and bio-medical information retrieval (TREC-COVID). Performance across these varied domains confirms that the dual-stage approach produces genuinely domain-agnostic ranking capabilities rather than overfitting to the MS MARCO training distribution.
The primary motivation for distillation is the latency-accuracy tradeoff. Consider the options for re-ranking 100 documents for a single query:
The 3B student represents the best latency-accuracy tradeoff for most production scenarios: it is 10–50x faster than GPT-4 while retaining most of the accuracy gains from distillation.
With 4-bit quantization (NF4), the 8B student requires approximately 5–6 GB of GPU memory, making it deployable on consumer-grade GPUs (RTX 3090, RTX 4090) or cost-effective cloud instances. The 3B student requires approximately 2–3 GB. Both LoRA adapters add negligible memory overhead (typically less than 100 MB combined), since LoRA parameters are a tiny fraction of the base model’s parameters.
For high-throughput deployments, the pointwise Stage 1 can be scaled horizontally with standard model serving frameworks (vLLM, TGI) since each query-document pair is independent. Stage 2 is inherently sequential (it processes one candidate list at a time) but processes only the top-K candidates, limiting its computational burden. In practice, the first-stage retriever (BM25 or dense) remains the throughput bottleneck at scale, not the re-ranker.
set_active_adapter() calls, and consider adding assertions in production code to validate the output format.