# Decomposing Retrieval Failures in RAG for Long-Document Financial Question Answering

Source: [Decomposing Retrieval Failures in RAG for Long-Document Financial Question Answering](<https://www.sota2.com/research/paper/699bded809c33900fcd80310>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2026-02-24T05:00:04.779Z

Authors: Amine Kobeissi, Philippe Langlais

Published: 2026-02-20T04:31:40.000Z

## Abstract

Retrieval-augmented generation is increasingly used for financial question answering over long regulatory filings, yet reliability depends on retrieving the exact context needed to justify answers in high stakes settings. We study a frequent failure mode in which the correct document is retrieved but the page or chunk that contains the answer is missed, leading the generator to extrapolate from incomplete context. Despite its practical significance, this within-document retrieval failure mode has received limited systematic attention in the Financial Question Answering (QA) literature. We evaluate retrieval at multiple levels of granularity, document, page, and chunk level, and introduce an oracle based analysis to provide empirical upper bounds on retrieval and generative performance. On a 150 question subset of FinanceBench, we reproduce and compare diverse retrieval strategies including dense, sparse, hybrid, and hierarchical methods with reranking and query reformulation. Across methods, gains in document discovery tend to translate into stronger page recall, yet oracle performance still suggests headroom for page and chunk level retrieval. To target this gap, we introduce a domain fine-tuned page scorer that treats pages as an intermediate retrieval unit between documents and chunks. Unlike prior passage-based hierarchical retrieval, we fine-tune a bi-encoder specifically for page-level relevance on financial filings, exploiting the semantic coherence of pages. Overall, our results demonstrate a significant improvement in page recall and chunk retrieval.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Information Retrieval on FinanceBench 150 samples](<https://www.sota2.com/research/sota/information-retrieval-on-financebench-150-samples.md>): Information Retrieval; FinanceBench 150 samples; DocRec@5: 95.

- [Financial Question Answering on FinanceBench (test)](<https://www.sota2.com/research/sota/financial-question-answering-on-financebench-test.md>): Financial Question Answering; FinanceBench (test); ROUGE-L: 15.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2602.17981>)

[PDF](<https://arxiv.org/pdf/2602.17981.pdf>)