# FinRAG-12B: A Production-Validated Recipe for Grounded Question Answering in Banking Source: [FinRAG-12B: A Production-Validated Recipe for Grounded Question Answering in Banking]() Collection index: [llms.txt]() Data updated: 2026-07-23T05:00:13.404Z Authors: Denys Katerenchuk, Pablo Duboue, Keelan Evanini, David Gondek, Nithin Govindugari, Olivier Allauzen, Joshua Baptiste, David J More, Joshua Schechter Published: 2026-05-06T22:04:44.000Z ## Abstract Large language models (LLMs) are rapidly being adopted across various domains. However, their adoption in banking industry faces resistance due to demands for high accuracy, regulatory compliance, and the need for verifiable and grounded responses. We present a unified, data-efficient framework for training grounded domain-specific LLMs that optimizes answer quality, citation grounding, and calibrated refusal under real-world deployment constraints. First, we describe a data generation pipeline that combines LLM-as-a-Judge filtering, citation annotation, and curriculum learning with only 143M tokens. The resulting 12B model achieves high answer quality outperforming GPT-4.1 on citation grounding, with a modest citation tradeoff versus the untuned base. Second, we propose a calibrated refusal mechanism: training on 22% unanswerable examples yield a 12% "I don't know" rate, substantially improving over the base model's unsafe 4.3% rate while avoiding GPT-4.1's over-refusal (20.2%). Third, we present an end-to-end methodology spanning from data curation to quantized serving. The system is deployed at 40+ financial institutions, achieving a 7.1 percentage point improvement in query resolution (p \< 0.001). Additionally, the model delivers 3-5x faster responses at 20-50x lower cost compared to GPT-4.1. ## Recorded benchmark results A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage. - [Question Answering on 258 banking QA examples (test)](): Question Answering; 258 banking QA examples (test); JudgeLM Score: 6.21. - [Question Answering on FinanceBench (F1 and Citation Rate)](): Question Answering; FinanceBench; F1 Score: 28.4. - [Banking Retrieval-Augmented Generation on U.S. Credit Union Queries May-December 2025 (production)](): Banking Retrieval-Augmented Generation (RAG); U.S. Credit Union Queries May-December 2025 2025-05-12 (production); Resolution Rate: 84.5. ## Original sources and implementation [Original paper]() [PDF]()