# Retrieval Head Mechanistically Explains Long-Context Factuality Source: [Retrieval Head Mechanistically Explains Long-Context Factuality]() Collection index: [llms.txt]() Data updated: 2025-10-26T02:38:38.445Z Authors: Wenhao Wu, Yizhong Wang, Guangxuan Xiao, Hao Peng, Yao Fu Published: 2024-04-24T00:24:03.000Z ## Abstract Despite the recent progress in long-context language models, it remains elusive how transformer-based models exhibit the capability to retrieve relevant information from arbitrary locations within the long context. This paper aims to address this question. Our systematic investigation across a wide spectrum of models reveals that a special type of attention heads are largely responsible for retrieving information, which we dub retrieval heads. We identify intriguing properties of retrieval heads:(1) universal: all the explored models with long-context capability have a set of retrieval heads; (2) sparse: only a small portion (less than 5\\%) of the attention heads are retrieval. (3) intrinsic: retrieval heads already exist in models pretrained with short context. When extending the context length by continual pretraining, it is still the same set of heads that perform information retrieval. (4) dynamically activated: take Llama-2 7B for example, 12 retrieval heads always attend to the required information no matter how the context is changed. The rest of the retrieval heads are activated in different contexts. (5) causal: completely pruning retrieval heads leads to failure in retrieving relevant information and results in hallucination, while pruning random non-retrieval heads does not affect the model's retrieval ability. We further show that retrieval heads strongly influence chain-of-thought (CoT) reasoning, where the model needs to frequently refer back the question and previously-generated context. Conversely, tasks where the model directly generates the answer using its intrinsic knowledge are less impacted by masking out retrieval heads. These observations collectively explain which internal part of the model seeks information from the input tokens. We believe our insights will foster future research on reducing hallucination, improving reasoning, and compressing the KV cache. ## Recorded benchmark results A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage. - [Passage Reranking on FEVER BEIR](): Passage Reranking; FEVER BEIR; NDCG@10: 72.59. - [Passage Reranking on ArguAna BEIR](): Passage Reranking; Arguana BEIR; NDCG@10: 33.87. - [Passage Reranking on FiQA BEIR](): Passage Reranking; FiQA BEIR; NDCG@10: 26.29. - [Passage Reranking on SciDocs BEIR](): Passage Reranking; Scidocs BEIR; NDCG@10: 15.44. - [Passage Reranking on News BEIR](): Passage Reranking; News BEIR; NDCG@10: 41.22. - [Passage Reranking on NQ BEIR](): Passage Reranking; NQ BEIR; NDCG@10: 35.98. - [Passage Reranking on COVID BEIR](): Passage Reranking; COVID BEIR; NDCG@10: 63.64. - [Passage Reranking on NFCorpus BEIR](): Passage Reranking; NFC BEIR; NDCG@10: 33.42. - [Passage Reranking on TREC Deep Learning 2020](): Passage Reranking; TREC Deep Learning 2020; NDCG@10: 43.25. - [Passage Reranking on DBPedia BEIR](): Passage Reranking; DBPedia BEIR; NDCG@10: 32.15. - [Passage Reranking on TREC Deep Learning 2019](): Passage Reranking; TREC Deep Learning 2019; NDCG@10: 48.98. ## Original sources and implementation [Original paper]() [PDF]()