# HeadRank: Decoding-Free Passage Reranking via Preference-Aligned Attention Heads

Source: [HeadRank: Decoding-Free Passage Reranking via Preference-Aligned Attention Heads](<https://www.sota2.com/research/paper/69e704cc7d0172140ec7efac>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2026-05-21T05:00:05.413Z

Authors: Juyuan Wang, Chenxing Wang, Yuchen Fang, Huiyun Hu, Junwu Du, Aolin Li, Shunlin Rong, Haijun Wu, Jin Xu, Ligang Liu, Dongliang Liao

Published: 2026-04-19T03:43:42.000Z

## Abstract

Decoding-free reranking methods that read relevance signals directly from LLM attention weights offer significant latency advantages over autoregressive approaches, yet suffer from attention score homogenization: middle-context documents receive near-identical scores, destroying the fine-grained distinctions required for ranking. We propose HeadRank, a framework that lifts preference optimization from discrete token space into the continuous attention domain through entropy-regularized head selection, hard adjacent-level preference pairs, and a distribution regularizer that jointly sharpen discriminability in the homogenized middle zone. Depth truncation at the deepest selected layer further reduces inference to $\\mathcal{O}(1)$ forward passes. Across 14 benchmarks on three Qwen3 scales (0.6B--4B) using only 211 training queries, HeadRank achieves the highest average NDCG@10 at every scale, outperforming both generative and decoding-free baselines on the majority of benchmarks with 100\\% formatting success. At 4B, 57.4\\% of relevant middle-zone documents reach the top quartile versus 14.2\\% for irrelevant ones -- a 43-percentage-point selectivity gap that demonstrates the effectiveness of attention-space preference alignment for listwise reranking.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Passage Reranking on TREC Deep Learning 2019](<https://www.sota2.com/research/sota/passage-reranking-on-trec-deep-learning-2019.md>): Passage Reranking; TREC Deep Learning 2019; NDCG@10: 64.33.

- [Passage Reranking on TREC Deep Learning 2020](<https://www.sota2.com/research/sota/passage-reranking-on-trec-deep-learning-2020.md>): Passage Reranking; TREC Deep Learning 2020; NDCG@10: 59.89.

- [Passage Reranking on COVID BEIR](<https://www.sota2.com/research/sota/passage-reranking-on-covid-beir.md>): Passage Reranking; COVID BEIR; NDCG@10: 73.94.

- [Passage Reranking on ArguAna BEIR](<https://www.sota2.com/research/sota/passage-reranking-on-arguana-beir.md>): Passage Reranking; Arguana BEIR; NDCG@10: 36.59.

- [Passage Reranking on SciDocs BEIR](<https://www.sota2.com/research/sota/passage-reranking-on-scidocs-beir.md>): Passage Reranking; Scidocs BEIR; NDCG@10: 16.96.

- [Passage Reranking on NFCorpus BEIR](<https://www.sota2.com/research/sota/passage-reranking-on-nfcorpus-beir.md>): Passage Reranking; NFC BEIR; NDCG@10: 36.44.

- [Passage Reranking on NQ BEIR](<https://www.sota2.com/research/sota/passage-reranking-on-nq-beir.md>): Passage Reranking; NQ BEIR; NDCG@10: 45.36.

- [Passage Reranking on FEVER BEIR](<https://www.sota2.com/research/sota/passage-reranking-on-fever-beir.md>): Passage Reranking; FEVER BEIR; NDCG@10: 73.47.

- [Multi-hop Reasoning on HotpotQA (Recall@2, Recall@5, SR)](<https://www.sota2.com/research/sota/multi-hop-reasoning-on-hotpotqa-recall-at-2-recall-at-5-sr.md>): Multi-hop Reasoning; HotpotQA; Recall@2: 72.65.

- [Multi-hop Reasoning on MuSiQue (Recall@2, Recall@5, SR)](<https://www.sota2.com/research/sota/multi-hop-reasoning-on-musique-recall-at-2-recall-at-5-sr.md>): Multi-hop Reasoning; MuSiQue; Recall@2: 42.67.

- [Passage Reranking on News BEIR](<https://www.sota2.com/research/sota/passage-reranking-on-news-beir.md>): Passage Reranking; News BEIR; NDCG@10: 46.4.

- [Passage Reranking on FiQA BEIR](<https://www.sota2.com/research/sota/passage-reranking-on-fiqa-beir.md>): Passage Reranking; FiQA BEIR; NDCG@10: 28.73.

- [Passage Reranking on DBPedia BEIR](<https://www.sota2.com/research/sota/passage-reranking-on-dbpedia-beir.md>): Passage Reranking; DBPedia BEIR; NDCG@10: 37.24.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2604.17237>)

[PDF](<https://arxiv.org/pdf/2604.17237.pdf>)