Question Answering on SQuAD (test)
91.2F1Human
Evaluation results
| Method | Links | ||||||
|---|---|---|---|---|---|---|---|
| Human2016.11 | 91.2 | 82.3 | — | — | — | — | |
| LoRAFederated Setting=cross-silo FL, Backbone=LLaMA2-13B, Number of Parameters=6.55M, Rank=82024.10 | 91.07 | — | — | — | — | — | |
| LoRAFederated Setting=large-scale FL, Backbone=LLaMA2-7B, Number of Parameters=4.19M, Rank=82024.10 | 90.49 | — | — | — | — | — | |
| FedTT+Federated Setting=cross-silo FL, Backbone=LLaMA2-13B, Number of Parameters=0.18M2024.10 | 90.4 | — | — | — | — | — | |
| FedTTFederated Setting=cross-silo FL, Backbone=LLaMA2-13B, Number of Parameters=0.64M2024.10 | 90.06 | — | — | — | — | — | |
| SWEPBackbone=BERT-base-uncased2021.05 | 89.43 | 82.24 | — | — | — | — | |
| CriSPOModel=Claude Instant, Shots=5-shot2024.10 | 89.4 | — | — | — | — | — | |
| CriSPOLLM=Claude Instant, shot_count=5-shot2024.10 | 89.4 | — | — | — | — | — | |
| FedTTFederated Setting=large-scale FL, Backbone=LLaMA2-7B, Number of Parameters=0.52M2024.10 | 89.32 | — | — | — | — | — | |
| Prior-AugBackbone=BERT-base-uncased2021.05 | 89.04 | 81.77 | — | — | — | — | |
| Gaussian-DropoutBackbone=BERT-base-uncased2021.05 | 88.78 | 81.47 | — | — | — | — | |
| Bernoulli-DropoutBackbone=BERT-base-uncased2021.05 | 88.76 | 81.46 | — | — | — | — | |
| Adv-AugBackbone=BERT-base-uncased2021.05 | 88.71 | 81.39 | — | — | — | — | |
| MLEBackbone=BERT-base-uncased2021.05 | 88.62 | 81.32 | — | — | — | — | |
| Word-DropoutBackbone=BERT-base-uncased2021.05 | 88.21 | 81.03 | — | — | — | — | |
| CriSPOModel=Claude Sonnet, Shots=5-shot2024.10 | 87.9 | — | — | — | — | — | |
| CriSPOLLM=Claude Sonnet, shot_count=5-shot2024.10 | 87.9 | — | — | — | — | — | |
| CriSPOModel=Claude Instant, Shots=0-shot2024.10 | 87.8 | — | — | — | — | — | |
| CriSPOLLM=Claude Instant, shot_count=0-shot2024.10 | 87.8 | — | — | — | — | — | |
| BiDAF + Self Attention + ELMoensemble=true2018.02 | 87.4 | 81 | — | — | — | — | |
| SSMBABackbone=BERT-base-uncased2021.05 | 86.53 | 78.17 | — | — | — | — | |
| Interactive AoA Reader+ensemble=true2018.02 | 86.5 | 79.1 | — | — | — | — | |
| DCN+ensemble=true2018.02 | 86 | 78.9 | — | — | — | — | |
| FusionNetensemble=true2018.02 | 86 | 79 | — | — | — | — | |
| RAdaZOModel Backbone=Ministral-3 14B, LoRA Rank=16, Best η=4e−52026.05 | 85.99 | — | — | — | — | — | |
| SWEPBackbone=ELECTRA-small-uncased2021.05 | 85.86 | 77.78 | — | — | — | — | |
| SSMBABackbone=ELECTRA-small-uncased2021.05 | 85.81 | 77.75 | — | — | — | — | |
| BiDAF + Self Attention + ELMoensemble=false2018.02 | 85.8 | 78.6 | — | — | — | — | |
| ELMo + BaselineModel Configuration=ELMo enhanced2018.02 | 85.8 | — | — | — | — | — | |
| MEAZOModel Backbone=Ministral-3 14B, LoRA Rank=16, Best η=2e−42026.05 | 85.75 | — | — | — | — | — | |
| Prior-AugBackbone=ELECTRA-small-uncased2021.05 | 85.6 | 77.7 | — | — | — | — | |
| ZO-AdamModel Backbone=Ministral-3 14B, LoRA Rank=16, Best η=2e−42026.05 | 85.41 | — | — | — | — | — | |
| In-context (Oracle)Backbone=QWEN2.5-7B, Setting=Oracle access to ground-truth context2026.05 | 85.4 | 78.2 | — | — | — | — | |
| CriSPOModel=Claude Sonnet, Shots=0-shot2024.10 | 85.3 | — | — | — | — | — | |
| CriSPOLLM=Claude Sonnet, shot_count=0-shot2024.10 | 85.3 | — | — | — | — | — | |
| ZO-SGDModel Backbone=Ministral-3 14B, LoRA Rank=16, Best η=5e−52026.05 | 85.29 | — | — | — | — | — | |
| MLEBackbone=ELECTRA-small-uncased2021.05 | 84.92 | 76.95 | — | — | — | — | |
| Gaussian-DropoutBackbone=ELECTRA-small-uncased2021.05 | 84.53 | 76.42 | — | — | — | — | |
| Bernoulli-DropoutBackbone=ELECTRA-small-uncased2021.05 | 84.5 | 76.31 | — | — | — | — | |
| SANensemble=false2018.02 | 84.4 | 76.8 | — | — | — | — | |
| Liu et al. (2017)Model Configuration=Previous SOTA2018.02 | 84.4 | — | — | — | — | — | |
| Adv-AugBackbone=ELECTRA-small-uncased2021.05 | 84.4 | 75.81 | — | — | — | — | |
| r-netensemble=false2018.02 | 84.3 | 76.5 | — | — | — | — | |
| Word-DropoutBackbone=ELECTRA-small-uncased2021.05 | 84.19 | 75.81 | — | — | — | — | |
| FusionNetensemble=false2018.02 | 83.9 | 76 | — | — | — | — | |
| Reg-RaSoRensemble=false2018.02 | 83.3 | 75.8 | — | — | — | — | |
| DP-AggZO K = 16Track=FT, Privacy=DP ε= 6 MI= 0.68, Model size=6.7B2026.05 | 83.3 | — | — | — | — | — | |
| ManualModel=Claude Sonnet, Shots=5-shot2024.10 | 83.2 | — | — | — | — | — | |
| ManualLLM=Claude Sonnet, shot_count=5-shot2024.10 | 83.2 | — | — | — | — | — | |
| DCN+ensemble=false2018.02 | 83.1 | 75.1 | — | — | — | — | |
| DP-AggZO K = 16Track=FT, Privacy=DP ε= 2 MI= 0.33, Model size=6.7B2026.05 | 82.9 | — | — | — | — | — | |
| ManualModel=Claude Instant, Shots=5-shot2024.10 | 82.5 | — | — | — | — | — | |
| ManualLLM=Claude Instant, shot_count=5-shot2024.10 | 82.5 | — | — | — | — | — | |
| BiDAF + Self Attentionensemble=false2018.02 | 81.1 | 72.1 | — | — | — | — | |
| Our baselineModel Configuration=Baseline2018.02 | 81.1 | — | — | — | — | — | |
| SEDT-LSTMEvaluation Mode=Ensemble2017.03 | 80.84 | 73.02 | — | — | — | — | |
| SECT-LSTMEvaluation Mode=Ensemble2017.03 | 80.58 | 72.83 | — | — | — | — | |
| BiDAFEvaluation Mode=Ensemble2017.03 | 80.33 | 72.33 | — | — | — | — | |
| DPZero K = 1Track=FT, Privacy=DP ε= 6 MI= 0.68, Model size=6.7B2026.05 | 80.1 | — | — | — | — | — | |
| R-net2017.03 | 79.7 | 71.3 | — | — | — | — | |
| ManualModel=Claude Instant, Shots=0-shot2024.10 | 79.5 | — | — | — | — | — | |
| ManualLLM=Claude Instant, shot_count=0-shot2024.10 | 79.5 | — | — | — | — | — | |
| DrQAconfiguration=Document Reader Only2017.03 | 79 | 70 | — | — | — | — | |
| OPROModel=Claude Instant2024.10 | 78.5 | — | — | — | — | — | |
| OPROLLM=Claude Instant2024.10 | 78.5 | — | — | — | — | — | |
| DPZero K = 1Track=FT, Privacy=DP ε= 2 MI= 0.33, Model size=6.7B2026.05 | 78.5 | — | — | — | — | — | |
| SEDT-LSTMEvaluation Mode=Single2017.03 | 77.97 | 68.48 | — | — | — | — | |
| DP-AggZO K = 16Track=FT, Privacy=DP ε= 6 MI= 0.68, Model size=1.3B2026.05 | 77.7 | — | — | — | — | — | |
| BiDAF2017.03 | 77.3 | 68 | — | — | — | — | |
| BiDAFensemble=false2018.02 | 77.3 | 68 | — | — | — | — | |
| SECT-LSTMEvaluation Mode=Single2017.03 | 77.21 | 68.12 | — | — | — | — | |
| BiDAFEvaluation Mode=Single2017.03 | 77.07 | 67.69 | — | — | — | — | |
| OPROModel=Claude Sonnet2024.10 | 76.4 | — | — | — | — | — | |
| OPROLLM=Claude Sonnet2024.10 | 76.4 | — | — | — | — | — | |
| DP-AggZO K = 16Track=FT, Privacy=DP ε= 2 MI= 0.33, Model size=1.3B2026.05 | 76.3 | — | — | — | — | — | |
| ManualModel=Claude Sonnet, Shots=0-shot2024.10 | 76.1 | — | — | — | — | — | |
| ManualLLM=Claude Sonnet, shot_count=0-shot2024.10 | 76.1 | — | — | — | — | — | |
| Dynamic Coattention Networks2017.03 | 75.9 | 66.2 | — | — | — | — | |
| RASORRepresentation=Recurrent span representations2016.11 | 75.5 | 67.4 | — | — | — | — | |
| MQANEvaluation mode=Single task scores2019.09 | 75.5 | — | — | — | — | — | |
| Multi-Perspective Matching2017.03 | 75.1 | 65.5 | — | — | — | — | |
| GNRQA Model=DCN+2018.05 | 75 | 66.6 | — | — | — | — | |
| DPZero K = 1Track=FT, Privacy=DP ε= 6 MI= 0.68, Model size=1.3B2026.05 | 74.2 | — | — | — | — | — | |
| RAGBackbone=QWEN2.5-7B, k=52026.05 | 73.37 | 65.18 | — | — | — | — | |
| FG fine-grained gate + ensembleintegration=Fine-grained Gating, ensemble=true2016.11 | 73.3 | 62.5 | — | — | — | — | |
| PACZERO-MITrack=FT, Privacy=DP ε= 2 MI= 0.33, Model size=6.7B2026.05 | 72.41 | — | — | — | — | — | |
| GPT-2Backbone=Smallest pre-trained GPT-2 model, Training epochs=9, Decoding=Greedy2019.09 | 72.3 | — | — | — | — | — | |
| DPZero K = 1Track=FT, Privacy=DP ε= 2 MI= 0.33, Model size=1.3B2026.05 | 72.3 | — | — | — | — | — | |
| RAGBackbone=QWEN2.5-7B, k=32026.05 | 72.18 | 63.89 | — | — | — | — | |
| PACZERO-ZPLTrack=FT, Privacy=I = 0, Model size=6.7B2026.05 | 72.12 | — | — | — | — | — | |
| ATACompressorBackbone=LLaMA-2-7B2026.02 | 71.67 | 53 | — | — | 27.18 | 3.14 | |
| PACZERO-MITrack=FT, Privacy=DP ε= 6 MI= 0.68, Model size=6.7B2026.05 | 71.64 | — | — | — | — | — | |
| DCR2016.10 | 71 | 62.5 | — | — | — | — | |
| Yu et al. (2016)2016.11 | 71 | 62.5 | — | — | — | — | |
| ATACompressorBackbone=Qwen-2-7B2026.02 | 70.52 | 52.1 | — | — | 27.39 | 3.07 | |
| Wang 20162016.10 | 70.3 | 59.5 | — | — | — | — | |
| Wang & Jiang (2016)2016.11 | 70.3 | 59.5 | — | — | — | — | |
| Match-LSTMVariant=Boundary2016.11 | 70 | 59.4 | — | — | — | — | |
| 500CompressorBackbone=LLaMA-2-7B2026.02 | 69.63 | 50.62 | — | — | 21.48 | 3.12 | |
| Original-ContextBackbone=LLaMA-2-7B2026.02 | 68.89 | 50.38 | — | — | 1 | 2.12 |