Retrieval-augmented generation often retrieves the right document but still sends too much of it to the generator. A top-k retriever may return ten long passages, yet only a handful of sentences directly answer the user query. The extra text increases token cost, raises latency, and gives the language model more distractors to reconcile.
Semantic context pruning inserts a lightweight relevance model between retrieval and generation. Instead of deciding whether an entire chunk should be kept, the pruner scores tokens, sentences, or spans inside each retrieved context and keeps only the parts that answer the query. The source article frames this as semantic highlighting: the system identifies meaningfully relevant sentences even when the exact query terms do not appear.
The important distinction from ordinary reranking is granularity. A reranker selects among chunks; a pruner compresses within chunks. It can therefore preserve evidence from several retrieved documents while removing irrelevant paragraphs, navigation text, repeated boilerplate, and misleading distractors before prompt assembly.
A semantic pruner treats context pruning as a sequence-labeling problem. The model receives the query and context together, produces a relevance score for each context token, and aggregates those token scores into sentence or span scores. This resembles extractive QA in that the output is grounded in the source text, but the goal is not one answer span; the goal is a compact evidence set.
Encoder-only reranker backbones are a natural fit because they already compare a query with candidate text. A pruning head changes the output shape from one scalar relevance score per document to many local scores across the context. The Zilliz implementation builds on the Provence-style approach and focuses training on the pruning head rather than optimizing a separate rerank head for this task.
High-quality pruning labels are expensive because annotators must mark the sentences that actually answer a query, including edge cases with distractors. The source describes generating training data with a reasoning-capable LLM that records its rationale while selecting relevant sentence spans. The rationale is not served at inference time; it helps create and debug better labels.
Semantic pruning is useful when retrieval recall is acceptable but context packing is wasteful. The system keeps the strongest evidence sentences from retrieved material, then passes a smaller and cleaner context to the generator.
The common inference path concatenates the query with a retrieved context, runs an encoder model, and obtains one score per context token. Sentence segmentation then converts dense token scores into application-friendly spans. A simple implementation can average token scores per sentence and keep sentences above a threshold or within a token budget.
def prune_context(model, tokenizer, query, context, max_tokens=900, threshold=0.55):
sentences = split_into_sentences(context)
spans = sentence_token_spans(tokenizer, query, context, sentences)
batch = tokenizer(query, context, return_tensors="pt", truncation=True)
token_scores = model(**batch).logits.sigmoid()[0]
ranked = []
for sentence, (start, end) in zip(sentences, spans):
score = token_scores[start:end].mean().item()
if score >= threshold:
ranked.append({"text": sentence, "score": score})
ranked.sort(key=lambda row: row["score"], reverse=True)
return pack_sentences(ranked, max_tokens=max_tokens)Production implementations need exact offset mapping from model tokens back to document text. They also need stable sentence splitting so the model’s predicted spans can be rendered, cited, and reused in logs.
The pruner should preserve source identifiers, document order, and enough surrounding context for pronouns and references to remain understandable. A sentence with a high score may depend on the previous sentence for the entity name, table label, or section heading. Practical systems often include a small neighbor window around selected sentences or merge adjacent selected spans.
def build_pruned_prompt(question, retrieved_docs):
evidence_blocks = []
for doc in retrieved_docs:
selected = prune_context(pruner, tokenizer, question, doc["text"])
if not selected:
continue
evidence_blocks.append({
"source": doc["source_id"],
"text": selected,
})
evidence = "\n\n".join(
f"[{block['source']}]\n{block['text']}" for block in evidence_blocks
)
return f"Answer using only the evidence below.\n\n{evidence}\n\nQuestion: {question}"A training example contains a query, a retrieved context, and labels for relevant spans or sentences. The source system builds bilingual data from public QA and web corpora, then uses an LLM annotation pipeline to identify supporting sentences. The model is trained to reproduce those relevance labels, producing a small inference-time component suitable for search-server deployment.
Semantic pruning directly reduces prompt size. Instead of sending every retrieved chunk to the generator, the system sends only high-value sentences. This is useful for long support articles, policy manuals, API documentation, legal contracts, and knowledge-base pages where retrieved chunks contain repeated headers, examples, or unrelated sections.
Agents often issue broad or decomposed retrieval queries during planning. The retrieved contexts can be noisy because the agent is exploring multiple hypotheses. A pruner can turn those broad hits into concise evidence packets before the agent reasons further, reducing the chance that irrelevant context steers the plan.
Because the output is a set of highlighted spans, engineers can inspect why a document was kept and which sentences were considered useful. This makes retrieval debugging easier than a black-box prompt that silently includes full chunks.
Use pruning as a budget allocator, not just a binary filter. Rank candidate spans by score, then pack them until the prompt budget is full. Preserve source diversity by limiting how many spans can come from a single document, especially when top retrieval results are near-duplicates.
def pack_with_source_caps(spans, token_budget, max_per_source=4):
kept, used, counts = [], 0, {}
for span in sorted(spans, key=lambda s: s["score"], reverse=True):
if counts.get(span["source"], 0) >= max_per_source:
continue
cost = estimate_tokens(span["text"])
if used + cost > token_budget:
continue
kept.append(span)
used += cost
counts[span["source"]] = counts.get(span["source"], 0) + 1
return restore_document_order(kept)After selecting a sentence, include adjacent headings, captions, or neighboring sentences when they resolve references. This guards against over-compression that keeps the answer but drops the context needed to interpret it correctly.
Combine a minimum relevance threshold with a final token budget. The threshold prevents very weak evidence from being included merely because budget remains. The budget prevents long documents with many medium-score sentences from dominating the prompt.