Reading Comprehension on SQuAD 2.0
93F1 ScoreFine-tuned SOTA
Evaluation results
| Method | Links | ||
|---|---|---|---|
| Fine-tuned SOTASetting=Fine-tuned2020.05 | 93 | — | |
| PaLM 540BEvaluation Protocol=Few-shot, Number of Shots=52022.04 | 83.3 | 79.6 | |
| PaLM 540BEvaluation Protocol=1-shot2022.04 | 82.9 | 78.7 | |
| PaLM 540BEvaluation Protocol=0-shot2022.04 | 80.8 | 75.5 | |
| Qwen3.5-9BShots=32026.04 | 79.25 | — | |
| Qwen3-14BShots=32026.04 | 74.4 | — | |
| HiLightSolver=Qwen3-14B, Actor model=Qwen3-0.6B, Highlight budget (γ)=0.152026.04 | 72.1 | 66.1 | |
| Llama-3.3-70B-InstructShots=32026.04 | 71.85 | — | |
| GLaM 62B/64EEvaluation Protocol=1-shot2022.04 | 71.8 | 66.5 | |
| GLaM 62B/64EEvaluation Protocol=Few-shot, Number of Shots=102022.04 | 71.8 | 67 | |
| SecGPT-14BShots=32026.04 | 71.8 | — | |
| GLaM 62B/64EEvaluation Protocol=0-shot2022.04 | 71.1 | 64.7 | |
| MALLM (Ranked Voting)Backbone=Llama 3 70B, Discussion Paradigm=Memory, Decision Protocol=Ranked Voting2026.07 | 70.6 | — | |
| MALLM (Average Voting)Backbone=Llama 3 70B, Discussion Paradigm=Memory2026.07 | 69.9 | — | |
| GPT-3Setting=Few-Shot2020.05 | 69.8 | — | |
| MALLM (Cumulative Voting)Backbone=Llama 3 70B, Discussion Paradigm=Memory, Decision Protocol=Cumulative Voting2026.07 | 69.7 | — | |
| Qwen3-8BShots=32026.04 | 69.6 | — | |
| BaselineBackbone=Llama 3 70B2026.07 | 69.5 | — | |
| MALLM (Simple Voting)Backbone=Llama 3 70B, Discussion Paradigm=Memory, Decision Protocol=Simple Voting2026.07 | 69.5 | — | |
| BFRSSolver=Qwen3-14B2026.04 | 69.4 | 57.3 | |
| DSPy (MIPROv2)Solver=Qwen3-14B2026.04 | 68.5 | 59 | |
| Baseline with CoTBackbone=Llama 3 70B2026.07 | 68 | — | |
| PRLSolver=Qwen3-14B2026.04 | 67.3 | 64.4 | |
| OPROSolver=Qwen3-14B2026.04 | 67.2 | 62 | |
| APESolver=Qwen3-14B2026.04 | 66.9 | 59.2 | |
| MALLM (Judge)Backbone=Llama 3 70B, Discussion Paradigm=Memory, Decision Protocol=Judge2026.07 | 65.7 | — | |
| GPT-3Setting=One-Shot2020.05 | 65.4 | — | |
| MISolver=Qwen3-14B2026.04 | 64.7 | 58.6 | |
| GPT-3Setting=Zero-Shot2020.05 | 59.5 | — | |
| MALLM (Majority Consensus)Backbone=Llama 3 70B, Discussion Paradigm=Memory, Decision Protocol=Majority Consensus2026.07 | 58.2 | — | |
| MALLM (Unanimity Consensus)Backbone=Llama 3 70B, Discussion Paradigm=Memory, Decision Protocol=Unanimity Consensus2026.07 | 56.7 | — | |
| MALLM (Average Consensus)Backbone=Llama 3 70B, Discussion Paradigm=Memory2026.07 | 56.4 | — | |
| XekRung-8BShots=32026.04 | 56 | — | |
| MALLM (Supermajority Consensus)Backbone=Llama 3 70B, Discussion Paradigm=Memory, Decision Protocol=Supermajority Consensus2026.07 | 54.3 | — | |
| Foundation-Sec-8B-ReasoningShots=32026.04 | 46.3 | — | |
| Llama-Primus-Reasoning-8BShots=32026.04 | 38.55 | — | |
| Llama-3.1-8B-InstructShots=32026.04 | 29.1 | — | |
| MALLM (Approval Voting)Backbone=Llama 3 70B, Discussion Paradigm=Memory, Decision Protocol=Approval Voting2026.07 | 24.3 | — | |
| PaLMprompting=1-shot2023.05 | — | 78.7 | |
| PaLM 2-Lprompting=1-shot2023.05 | — | 80.5 | |
| PaLM 2-Mprompting=1-shot2023.05 | — | 77.1 | |
| PaLM 2-Sprompting=1-shot2023.05 | — | 75.7 |