Multiple Choice Question Answering on OBQA
93.2AccuracyETN
Evaluation results
| Method | Links | ||||
|---|---|---|---|---|---|
| ETNBackbone=Gemma-2-9B2026.04 | 93.2 | — | 98.06 | 96 | |
| DMMBackbone=Gemma-2-9B2026.04 | 93 | — | 96.68 | 95.44 | |
| LLBackbone=Gemma-2-9B2026.04 | 92.53 | — | 79.45 | — | |
| MCDBackbone=Gemma-2-9B2026.04 | 89.6 | — | 97.87 | — | |
| DeepEnsBackbone=Gemma-2-9B2026.04 | 89 | — | 97.84 | — | |
| MAPCEBackbone=Gemma-2-9B2026.04 | 88.87 | — | 97.59 | — | |
| QAPBackbone=Flan-T5, Parameter Count=11B2025.03 | 87.74 | 0.43 | — | — | |
| MAPEDLBackbone=Gemma-2-9B2026.04 | 87.53 | — | 94.63 | 86.32 | |
| IB-EDLBackbone=Gemma-2-9B2026.04 | 85.13 | — | 94.6 | 92.15 | |
| GNPBackbone=Flan-T5, Parameter Count=11B2025.03 | 85.04 | 0.32 | — | — | |
| SPBackbone=Flan-T5, Parameter Count=11B2025.03 | 84.36 | 0.34 | — | — | |
| PEBackbone=Flan-T5, Parameter Count=11B2025.03 | 82.98 | 0.21 | — | — | |
| QAPBackbone=Flan-T5, Parameter Count=3B2025.03 | 81.62 | 0.44 | — | — | |
| LLMBackbone=Flan-T5, Parameter Count=11B2025.03 | 80.12 | 0.19 | — | — | |
| KGEPBackbone=Flan-T5, Parameter Count=11B2025.03 | 77.6 | 0.4 | — | — | |
| Similarity-based RouterSetting=Student, Backbone=FLAN-T5 large (783M)2026.02 | 76.6 | — | — | — | |
| GNPBackbone=Flan-T5, Parameter Count=3B2025.03 | 76.12 | 0.34 | — | — | |
| Teacher SelectionSetting=Student, Backbone=FLAN-T5 large (783M)2026.02 | 75.2 | — | — | — | |
| SPBackbone=Flan-T5, Parameter Count=3B2025.03 | 74.94 | 0.64 | — | — | |
| PEBackbone=Flan-T5, Parameter Count=3B2025.03 | 74.88 | 0.14 | — | — | |
| PLM ClassifierSetting=Student, Backbone=FLAN-T5 large (783M)2026.02 | 74.2 | — | — | — | |
| LLMBackbone=Flan-T5, Parameter Count=3B2025.03 | 73.4 | 0.14 | — | — | |
| KGEPBackbone=Flan-T5, Parameter Count=3B2025.03 | 72.72 | 0.3 | — | — | |
| Plackett-Luce RankingSetting=Student, Backbone=FLAN-T5 large (783M)2026.02 | 72.6 | — | — | — | |
| Fine-tuningSetting=Student, Backbone=FLAN-T5 large (783M)2026.02 | 72 | — | — | — | |
| Distilling-Step-by-StepSetting=Student, Backbone=FLAN-T5 large (783M)2026.02 | 71.6 | — | — | — | |
| Knowledge AggregationSetting=Student, Backbone=FLAN-T5 large (783M)2026.02 | 71.4 | — | — | — | |
| FLAN-T5 xlargeSetting=Teacher2026.02 | 69.2 | — | — | — | |
| TinyLLMSetting=Student, Backbone=FLAN-T5 large (783M)2026.02 | 68.8 | — | — | — | |
| QAPBackbone=Llama2-chat, Parameter Count=7B2025.03 | 67.52 | 0.45 | — | — | |
| GPT-4o2026.02 | 67.21 | — | — | — | |
| QAPBackbone=Llama2-chat, Parameter Count=13B2025.03 | 66.32 | 0.44 | — | — | |
| Llama-3.1-8B-InstructSetting=Teacher2026.02 | 65.6 | — | — | — | |
| Similarity-based RouterSetting=Student, Backbone=FLAN-T5 base (248M)2026.02 | 65.6 | — | — | — | |
| Teacher SelectionSetting=Student, Backbone=FLAN-T5 base (248M)2026.02 | 65 | — | — | — | |
| PLM ClassifierSetting=Student, Backbone=FLAN-T5 base (248M)2026.02 | 63.6 | — | — | — | |
| Plackett-Luce RankingSetting=Student, Backbone=FLAN-T5 base (248M)2026.02 | 62.4 | — | — | — | |
| Fine-tuningSetting=Student, Backbone=FLAN-T5 base (248M)2026.02 | 61 | — | — | — | |
| Distilling-Step-by-StepSetting=Student, Backbone=FLAN-T5 base (248M)2026.02 | 59 | — | — | — | |
| TinyLLMSetting=Student, Backbone=FLAN-T5 base (248M)2026.02 | 58.8 | — | — | — | |
| GNPBackbone=Llama2-chat, Parameter Count=13B2025.03 | 58.72 | 0.33 | — | — | |
| Mistral Large2026.02 | 58.66 | — | — | — | |
| Knowledge AggregationSetting=Student, Backbone=FLAN-T5 base (248M)2026.02 | 58 | — | — | — | |
| LLMBackbone=Llama2-chat, Parameter Count=7B2025.03 | 57.28 | 0.09 | — | — | |
| GNPBackbone=Llama2-chat, Parameter Count=7B2025.03 | 56.88 | 0.35 | — | — | |
| Llama 2-chatSetting=Teacher2026.02 | 54.6 | — | — | — | |
| KGEPBackbone=Llama2-chat, Parameter Count=7B2025.03 | 52.72 | 0.29 | — | — | |
| PEBackbone=Llama2-chat, Parameter Count=7B2025.03 | 52.48 | 0.1 | — | — | |
| KGEPBackbone=Llama2-chat, Parameter Count=13B2025.03 | 52 | 0.45 | — | — | |
| BioMistral-7BSetting=Teacher2026.02 | 51.8 | — | — | — | |
| LLMBackbone=Llama2-chat, Parameter Count=13B2025.03 | 51.04 | 0.1 | — | — | |
| PEBackbone=Llama2-chat, Parameter Count=13B2025.03 | 51 | 0.18 | — | — | |
| InferenceSetting=Student, Backbone=FLAN-T5 large (783M)2026.02 | 50.4 | — | — | — | |
| Similarity-based RouterSetting=Student, Backbone=FLAN-T5 small (77M)2026.02 | 48.6 | — | — | — | |
| Teacher SelectionSetting=Student, Backbone=FLAN-T5 small (77M)2026.02 | 46.8 | — | — | — | |
| Fine-tuningSetting=Student, Backbone=FLAN-T5 small (77M)2026.02 | 45.6 | — | — | — | |
| PLM ClassifierSetting=Student, Backbone=FLAN-T5 small (77M)2026.02 | 44.2 | — | — | — | |
| Claude 3 Haiku2026.02 | 42.32 | — | — | — | |
| Plackett-Luce RankingSetting=Student, Backbone=FLAN-T5 small (77M)2026.02 | 42 | — | — | — | |
| Distilling-Step-by-StepSetting=Student, Backbone=FLAN-T5 small (77M)2026.02 | 41 | — | — | — | |
| Llama3-70B-InstructSize=70B, Type=Instruct2026.02 | 40.94 | — | — | — | |
| TinyLLMSetting=Student, Backbone=FLAN-T5 small (77M)2026.02 | 40.6 | — | — | — | |
| Knowledge AggregationSetting=Student, Backbone=FLAN-T5 small (77M)2026.02 | 40.4 | — | — | — | |
| AR (autoregressive)Zero-shot=true, Likelihood-based evaluation=true, Number of Parameters=1.7B2026.05 | 40.4 | — | — | — | |
| AR (autoregressive, ours)Zero-shot=true, Likelihood-based evaluation=true, Number of Parameters=1.7B2026.05 | 40.2 | — | — | — | |
| Duo (uniform diffusion)Zero-shot=true, Likelihood-based evaluation=true, Number of Parameters=1.7B2026.05 | 33 | — | — | — | |
| MDLM (masked diffusion)Zero-shot=true, Likelihood-based evaluation=true, Number of Parameters=1.7B2026.05 | 32 | — | — | — | |
| InferenceSetting=Student, Backbone=FLAN-T5 base (248M)2026.02 | 31 | — | — | — | |
| Eso-LM (interpolating)Zero-shot=true, Likelihood-based evaluation=true, Number of Parameters=1.7B2026.05 | 29.6 | — | — | — | |
| SPBackbone=Llama2-chat, Parameter Count=13B2025.03 | 28.9 | 0.27 | — | — | |
| MDLMevaluation protocol=likelihood-based answer scoring2026.05 | 28.6 | — | — | — | |
| SPBackbone=Llama2-chat, Parameter Count=7B2025.03 | 27.16 | 0.45 | — | — | |
| SMDM-1BZero-shot=true, Likelihood-based evaluation=true, Number of Parameters=1B2026.05 | 27 | — | — | — | |
| ChanceZero-shot=true, Likelihood-based evaluation=true2026.05 | 26.6 | — | — | — | |
| CoFReevaluation protocol=likelihood-based answer scoring2026.05 | 25.2 | — | — | — | |
| CFM (ours)Zero-shot=true, Likelihood-based evaluation=true, Number of Parameters=1.7B2026.05 | 24.4 | — | — | — | |
| Qwen1.5 (1.8B)Size=1.8B2026.02 | 20.37 | — | — | — | |
| InferenceSetting=Student, Backbone=FLAN-T5 small (77M)2026.02 | 16.6 | — | — | — | |
| Gemma (2B)Size=2B2026.02 | 14.26 | — | — | — |