Multiple Choice Question Answering on RACE
98.24AccuracyQA
Evaluation Results
| Method | Links | |||
|---|---|---|---|---|
| QACoverage=20%2022.09 | 98.24 | — | — | |
| QACoverage=50%2022.09 | 93.88 | — | — | |
| QA+CCoverage=50%2022.09 | 92.68 | — | — | |
| ETNBackbone=Gemma-2-9B2026.04 | 89.48 | 98.43 | 95.94 | |
| DMMBackbone=Gemma-2-9B2026.04 | 89.42 | 97.96 | 95.2 | |
| MCDBackbone=Gemma-2-9B2026.04 | 89.38 | 98.4 | — | |
| DeepEnsBackbone=Gemma-2-9B2026.04 | 89.09 | 98.26 | — | |
| MAPCEBackbone=Gemma-2-9B2026.04 | 88.81 | 98.16 | — | |
| LLBackbone=Gemma-2-9B2026.04 | 88.51 | 69.05 | — | |
| IB-EDLBackbone=Gemma-2-9B2026.04 | 87.98 | 93.72 | 88.04 | |
| MAPEDLBackbone=Gemma-2-9B2026.04 | 87.14 | 92.27 | 87.6 | |
| QA+E+Cmodel=RoBERTa-RACE, entailment=true, contradiction=true2022.09 | 77.53 | — | — | |
| QA+Emodel=RoBERTa-RACE, entailment=true2022.09 | 77.18 | — | — | |
| QA+Cmodel=RoBERTa-RACE, contradiction=true2022.09 | 77.16 | — | — | |
| QAmodel=RoBERTa-RACE2022.09 | 76.93 | — | — | |
| Mistral Large2026.02 | 70.17 | — | — | |
| GPT-4o2026.02 | 67.87 | — | — | |
| OLTQAConfiguration=Full2023.05 | 59.65 | — | — | |
| ProQA2023.05 | 59.55 | — | — | |
| OLTQAKnowledge Distillation=Static MKD2023.05 | 59.55 | — | — | |
| OLTQARetriever=EPR2023.05 | 59.36 | — | — | |
| OLTQAAblation=without Pk2023.05 | 59.34 | — | — | |
| OLTQAKnowledge Distillation=without MKD2023.05 | 58.95 | — | — | |
| Hyperformer++2023.05 | 58.24 | — | — | |
| OLTQARetriever=BM252023.05 | 58.22 | — | — | |
| Llama3-70B-InstructSize=70B, Type=Instruct2026.02 | 58.21 | — | — | |
| OLTQAKnowledge Distillation=Back KD2023.05 | 58.17 | — | — | |
| OLTQAAblation=without Pm2023.05 | 58.13 | — | — | |
| OLTQARe-ranker=excluded2023.05 | 57.64 | — | — | |
| Claude 3 Haiku2026.02 | 57.2 | — | — | |
| Muppet2023.05 | 57.13 | — | — | |
| UnifiedQA2023.05 | 56.97 | — | — | |
| ScaleOTSetting=Plug-in, Backbone=OPT-1.3B2025.07 | 40.8 | — | — | |
| GradOTSetting=Plug-in, Backbone=OPT-1.3B2025.07 | 40.6 | — | — | |
| CRaShSetting=Plug-in, Backbone=OPT-1.3B2025.07 | 39.9 | — | — | |
| ScaleOTSetting=Emulator Fine-tuning (Emu. FT), Backbone=OPT-1.3B2025.07 | 39.1 | — | — | |
| OTSetting=Plug-in, Backbone=OPT-1.3B2025.07 | 38.9 | — | — | |
| CRaShSetting=Emulator Fine-tuning (Emu. FT), Backbone=OPT-1.3B2025.07 | 38.9 | — | — | |
| EPR2023.05 | 38.82 | — | — | |
| GradOTSetting=Emulator Fine-tuning (Emu. FT), Backbone=OPT-1.3B2025.07 | 38.7 | — | — | |
| OTSetting=Emulator Fine-tuning (Emu. FT), Backbone=OPT-1.3B2025.07 | 38.6 | — | — | |
| OT†Setting=Plug-in, Backbone=OPT-1.3B2025.07 | 37.9 | — | — | |
| Full LLMSetting=Fine-tuning (FT), Backbone=OPT-1.3B2025.07 | 37 | — | — | |
| OT†Setting=Emulator Fine-tuning (Emu. FT), Backbone=OPT-1.3B2025.07 | 36.5 | — | — | |
| AR (autoregressive, ours)Zero-shot=true, Likelihood-based evaluation=true, Number of Parameters=1.7B2026.05 | 36.3 | — | — | |
| AR (autoregressive)Zero-shot=true, Likelihood-based evaluation=true, Number of Parameters=1.7B2026.05 | 36.2 | — | — | |
| SciQ-smallvariant=small2022.09 | 35.56 | — | — | |
| Duo (uniform diffusion)Zero-shot=true, Likelihood-based evaluation=true, Number of Parameters=1.7B2026.05 | 35 | — | — | |
| Qwen1.5 (1.8B)Size=1.8B2026.02 | 34.91 | — | — | |
| MDLM (masked diffusion)Zero-shot=true, Likelihood-based evaluation=true, Number of Parameters=1.7B2026.05 | 34.7 | — | — | |
| Full LLMSetting=Zero-shot (ZS), Backbone=OPT-1.3B2025.07 | 34.2 | — | — | |
| CRaShSetting=Emulator Zero-shot (Emu. ZS), Backbone=OPT-1.3B2025.07 | 34.2 | — | — | |
| OTSetting=Emulator Zero-shot (Emu. ZS), Backbone=OPT-1.3B2025.07 | 33 | — | — | |
| SciQ-basevariant=base2022.09 | 30.97 | — | — | |
| SMDM-1BZero-shot=true, Likelihood-based evaluation=true, Number of Parameters=1B2026.05 | 29.3 | — | — | |
| CFM (ours)Zero-shot=true, Likelihood-based evaluation=true, Number of Parameters=1.7B2026.05 | 28.3 | — | — | |
| MDLMevaluation protocol=likelihood-based answer scoring2026.05 | 28.04 | — | — | |
| CoFReevaluation protocol=likelihood-based answer scoring2026.05 | 27.27 | — | — | |
| ScaleOTSetting=Emulator Zero-shot (Emu. ZS), Backbone=OPT-1.3B2025.07 | 27.2 | — | — | |
| Eso-LM (interpolating)Zero-shot=true, Likelihood-based evaluation=true, Number of Parameters=1.7B2026.05 | 26.1 | — | — | |
| ChanceZero-shot=true, Likelihood-based evaluation=true2026.05 | 24.2 | — | — | |
| OT†Setting=Emulator Zero-shot (Emu. ZS), Backbone=OPT-1.3B2025.07 | 22.7 | — | — | |
| GradOTSetting=Emulator Zero-shot (Emu. ZS), Backbone=OPT-1.3B2025.07 | 22.3 | — | — | |
| Gemma (2B)Size=2B2026.02 | 14.32 | — | — |