Numerical Question Answering on FinQA (test)
91.16Execution AccuracyHuman Expert Performance
Evaluation Results
| Method | Links | ||
|---|---|---|---|
| Human Expert Performance2021.09 | 91.16 | 87.49 | |
| Human Expert2022.10 | 91.16 | 87.49 | |
| FinQANet-Gold (RoBERTa-large)Input Source=Gold Retriever, Program Generator Backbone=RoBERTa-large2021.09 | 70 | 68.76 | |
| Multi-Retriever RAG (SecBERT + DPR-FAISS) + Gemini-1.5-pro-latest Generator + Few Shot PromptRetriever Type=Internal + External (Multi-Retriever), Prompting Strategy=Few Shot Prompt2025.12 | 69.37 | — | |
| ELASTICEncoder=RoBERTa-large2022.10 | 68.96 | 65.21 | |
| SecBERT Internal Retriever + Gemini-1.5-pro-latest Generator + Few Shot PromptRetriever Type=Internal, Prompting Strategy=Few Shot Prompt2025.12 | 66.02 | — | |
| FinQANetEncoder=RoBERTa-large2022.10 | 65.05 | 63.52 | |
| ELASTICEncoder=RoBERTa-base2022.10 | 62.66 | 59.28 | |
| FinQANet (RoBERTa-large)Retriever=BERT-base, Program Generator Backbone=RoBERTa-large2021.09 | 61.24 | 58.86 | |
| FinQANetEncoder=RoBERTa-base2022.10 | 60.1 | 58.38 | |
| FinQANet (RoBERTa-base)Retriever=BERT-base, Program Generator Backbone=RoBERTa-base2021.09 | 56.1 | 54.38 | |
| FinQANet (BERT-large)Retriever=BERT-base, Program Generator Backbone=BERT-large2021.09 | 53.52 | 51.62 | |
| NeRd2022.10 | 52.48 | 49.9 | |
| General Crowd Performance2021.09 | 50.68 | 48.17 | |
| Human Non-Expert2022.10 | 50.68 | 48.17 | |
| FinQANet (FinBert)Retriever=BERT-base, Program Generator Backbone=FinBert2021.09 | 50.1 | 47.52 | |
| FinQANet (BERT-base)Retriever=BERT-base, Program Generator Backbone=BERT-base2021.09 | 50 | 48 | |
| Retriever + NeRd (BERT-base)Retriever=BERT-base, Program Generator=NeRd, Backbone=BERT-base2021.09 | 48.57 | 46.76 | |
| Multi-Retriever RAG (SecBERT + DPR-FAISS) + Gemini-1.5-pro-latest GeneratorRetriever Type=Internal + External (Multi-Retriever), Prompting Strategy=Single Prompt2025.12 | 45.33 | — | |
| SecBERT Internal Retriever + Gemini-1.5-pro-latest GeneratorRetriever Type=Internal, Prompting Strategy=Single Prompt2025.12 | 41.84 | — | |
| Multi-Retriever RAG (SecBERT + DPR-FAISS) + Gemini-1.0-pro GeneratorRetriever Type=Internal + External (Multi-Retriever), Prompting Strategy=Single Prompt2025.12 | 41.75 | — | |
| SecBERT Internal Retriever + Gemini-1.0-pro GeneratorRetriever Type=Internal, Prompting Strategy=Single Prompt2025.12 | 39.3 | — | |
| Gemini-1.5-pro-latestRetriever Type=None, Prompting Strategy=Zero-shot2025.12 | 36.27 | — | |
| Gemini-1.0-proRetriever Type=None, Prompting Strategy=Zero-shot2025.12 | 34.76 | — | |
| Multi-Retriever RAG (SecBERT + DPR-FAISS) + Gemini-1.0-pro Generator + Few Shot PromptRetriever Type=Internal + External (Multi-Retriever), Prompting Strategy=Few Shot Prompt2025.12 | 24.69 | — | |
| SecBERT Internal Retriever + Gemini-1.0-pro Generator + Few Shot PromptRetriever Type=Internal, Prompting Strategy=Few Shot Prompt2025.12 | 22.03 | — | |
| Pre-Trained Longformer (base)Backbone=Longformer (base), Retriever=None (End-to-End)2021.09 | 21.9 | 20.48 | |
| Retriever + Seq2seqRetriever=BERT-base, Program Generator=Seq2seq2021.09 | 19.71 | 18.38 | |
| NumNet+2022.10 | 10.29 | — | |
| NumNet2022.10 | 2.32 | — | |
| TF-IDF + Single OpRetriever=TF-IDF, Program Generator=Single Op2021.09 | 1.01 | 0.9 | |
| Graph2Tree2022.10 | 0.37 | 0 | |
| Retriever + Direct GenerationRetriever=BERT-base, Program Generator=Direct Generation2021.09 | 0.3 | — |