In-context ranking asks a language model to rank a shortlist of candidate documents by placing the task instruction, the query, and all candidate documents into a single prompt. This can produce strong listwise relevance judgments because the model sees the whole candidate set at once, but it inherits the Transformer attention cost of long prompts. As the shortlist grows, ordinary full attention spends compute on document-to-document interactions that often do not help decide relevance to the query.
Blockwise In-Context Ranking, introduced in Google Research’s BlockRank work, makes this pattern more scalable by exploiting structure in the prompt. The key observation is that in fine-tuned in-context rankers, attention is usually dense inside each document block but sparse across different document blocks, while selected query-token attention patterns correlate with document relevance. BlockRank turns those observations into an attention layout and auxiliary training objective rather than treating the long prompt as one undifferentiated token sequence.
The result is a ranker that keeps the useful parts of listwise LLM ranking while reducing unnecessary attention. The paper reports that a Mistral-7B BlockRank model matched or exceeded strong listwise rankers on BEIR, MS MARCO, and Natural Questions, with a reported 4.7x inference speedup for 100 MS MARCO documents and graceful scaling to roughly 500 in-context documents.
In-context ranking differs from ordinary cross-encoder reranking. A cross-encoder scores one query-document pair at a time; an in-context ranker receives the query plus many candidate documents and emits a relevance decision or ranked list. This lets the model compare candidates in one prompt, but it also means the prompt can become extremely long after a first-stage retriever returns 50, 100, or 500 candidates.
The prompt can be partitioned into semantic regions: task instructions, query tokens, and one block per candidate document. Full attention allows every token in every document to attend to every token in every other document. BlockRank instead treats each candidate as a block and enforces a sparse attention pattern that preserves query-to-document and within-document attention while reducing cross-document attention.
BlockRank relies on the finding that attention from certain query tokens to document blocks in middle layers is predictive of actual relevance. Instead of only training the model to generate the right final answer, the method also trains internal attention to concentrate on truly relevant document blocks. This turns attention into a supervised retrieval signal.
A blockwise ranker first constructs a prompt where document boundaries are explicit. The attention mask permits dense attention inside each document block and between the query and document blocks, but it masks most document-to-document attention. This changes scaling from full quadratic attention over the entire concatenated context toward linear growth in the number of candidate documents, because each document is processed mostly independently except for shared query interaction.
def build_blockrank_mask(query_tokens, documents):
blocks = ["instruction", "query"] + [f"doc_{i}" for i in range(len(documents))]
mask = DenseFalseMatrix(total_tokens(blocks), total_tokens(blocks))
allow(mask, "instruction", "instruction")
allow(mask, "query", "instruction")
allow(mask, "query", "query")
for doc_id in document_blocks(blocks):
allow(mask, doc_id, doc_id) # dense within-document reading
allow(mask, doc_id, "query") # document can attend to query
allow(mask, "query", doc_id) # query can collect relevance evidence
return maskBlockRank fine-tuning combines the normal generative objective with an auxiliary objective over query-document block relevance. Positive documents should receive stronger attention from selected query tokens than negatives. This encourages the model’s internal attention maps to behave like a relevance detector, not just a generic next-token generator.
In practice, this means training examples need query, candidate documents, and labels identifying the relevant candidate or desired ranking. The model learns both how to output the ranking decision and how to allocate attention efficiently while doing so.
Blockwise in-context ranking fits after a fast first-stage retriever such as BM25, hybrid search, or ANN vector retrieval. The retriever produces a shortlist; the blockwise ranker consumes the shortlist in one prompt and returns the final ranking. This is useful when pairwise cross-encoders are too expensive or when listwise comparison improves quality.
Enterprise search often needs to compare many moderately long passages: policies, tickets, contracts, runbooks, and internal documentation. Block-sparse listwise ranking lets a system use an LLM’s semantic understanding without paying full attention over all cross-document combinations.
The production pattern is a cascade. Keep first-stage retrieval cheap and high-recall, then spend LLM compute only on a bounded shortlist. The ranker prompt should preserve stable document identifiers so the model can emit a ranked list that maps back to source records.
def rerank(query, corpus_index, ranker, k=100):
candidates = corpus_index.search(query, top_k=k)
prompt = format_rank_prompt(
instruction="Rank documents by relevance to the query.",
query=query,
documents=[c.text for c in candidates],
)
ranked_ids = ranker.generate(prompt, attention_layout="blockrank")
return [candidates[i] for i in ranked_ids]The serving runtime must support custom attention masks or an equivalent block-sparse kernel. If the inference stack silently falls back to dense attention, the method loses its main efficiency benefit. Teams should validate both output quality and actual kernel behavior with profiler traces.