List your company
ResearchFormerly WizWand

Multiple Choice Question Answering on RACE

98.24AccuracyQA
10.963233.621656.2878.9384Sep 28, 2022May 9, 2023Dec 18, 2023Jul 28, 2024Mar 8, 2025Oct 17, 2025May 29, 2026
Updated 5mo ago

Evaluation results

MethodLinks
QACoverage=20%2022.0998.24——
QACoverage=50%2022.0993.88——
QA+CCoverage=50%2022.0992.68——
ETNBackbone=Gemma-2-9B2026.0489.4898.4395.94
DMMBackbone=Gemma-2-9B2026.0489.4297.9695.2
MCDBackbone=Gemma-2-9B2026.0489.3898.4—
DeepEnsBackbone=Gemma-2-9B2026.0489.0998.26—
MAPCEBackbone=Gemma-2-9B2026.0488.8198.16—
LLBackbone=Gemma-2-9B2026.0488.5169.05—
IB-EDLBackbone=Gemma-2-9B2026.0487.9893.7288.04
MAPEDLBackbone=Gemma-2-9B2026.0487.1492.2787.6
QA+E+Cmodel=RoBERTa-RACE, entailment=true, contradiction=true2022.0977.53——
QA+Emodel=RoBERTa-RACE, entailment=true2022.0977.18——
QA+Cmodel=RoBERTa-RACE, contradiction=true2022.0977.16——
QAmodel=RoBERTa-RACE2022.0976.93——
Mistral Large2026.0270.17——
GPT-4o2026.0267.87——
OLTQAConfiguration=Full2023.0559.65——
ProQA2023.0559.55——
OLTQAKnowledge Distillation=Static MKD2023.0559.55——
OLTQARetriever=EPR2023.0559.36——
OLTQAAblation=without Pk2023.0559.34——
OLTQAKnowledge Distillation=without MKD2023.0558.95——
Hyperformer++2023.0558.24——
OLTQARetriever=BM252023.0558.22——
Llama3-70B-InstructSize=70B, Type=Instruct2026.0258.21——
OLTQAKnowledge Distillation=Back KD2023.0558.17——
OLTQAAblation=without Pm2023.0558.13——
OLTQARe-ranker=excluded2023.0557.64——
Claude 3 Haiku2026.0257.2——
Muppet2023.0557.13——
UnifiedQA2023.0556.97——
ScaleOTSetting=Plug-in, Backbone=OPT-1.3B2025.0740.8——
GradOTSetting=Plug-in, Backbone=OPT-1.3B2025.0740.6——
CRaShSetting=Plug-in, Backbone=OPT-1.3B2025.0739.9——
ScaleOTSetting=Emulator Fine-tuning (Emu. FT), Backbone=OPT-1.3B2025.0739.1——
OTSetting=Plug-in, Backbone=OPT-1.3B2025.0738.9——
CRaShSetting=Emulator Fine-tuning (Emu. FT), Backbone=OPT-1.3B2025.0738.9——
EPR2023.0538.82——
GradOTSetting=Emulator Fine-tuning (Emu. FT), Backbone=OPT-1.3B2025.0738.7——
OTSetting=Emulator Fine-tuning (Emu. FT), Backbone=OPT-1.3B2025.0738.6——
OT†Setting=Plug-in, Backbone=OPT-1.3B2025.0737.9——
Full LLMSetting=Fine-tuning (FT), Backbone=OPT-1.3B2025.0737——
OT†Setting=Emulator Fine-tuning (Emu. FT), Backbone=OPT-1.3B2025.0736.5——
AR (autoregressive, ours)Zero-shot=true, Likelihood-based evaluation=true, Number of Parameters=1.7B2026.0536.3——
AR (autoregressive)Zero-shot=true, Likelihood-based evaluation=true, Number of Parameters=1.7B2026.0536.2——
SciQ-smallvariant=small2022.0935.56——
Duo (uniform diffusion)Zero-shot=true, Likelihood-based evaluation=true, Number of Parameters=1.7B2026.0535——
Qwen1.5 (1.8B)Size=1.8B2026.0234.91——
MDLM (masked diffusion)Zero-shot=true, Likelihood-based evaluation=true, Number of Parameters=1.7B2026.0534.7——
Full LLMSetting=Zero-shot (ZS), Backbone=OPT-1.3B2025.0734.2——
CRaShSetting=Emulator Zero-shot (Emu. ZS), Backbone=OPT-1.3B2025.0734.2——
OTSetting=Emulator Zero-shot (Emu. ZS), Backbone=OPT-1.3B2025.0733——
SciQ-basevariant=base2022.0930.97——
SMDM-1BZero-shot=true, Likelihood-based evaluation=true, Number of Parameters=1B2026.0529.3——
CFM (ours)Zero-shot=true, Likelihood-based evaluation=true, Number of Parameters=1.7B2026.0528.3——
MDLMevaluation protocol=likelihood-based answer scoring2026.0528.04——
CoFReevaluation protocol=likelihood-based answer scoring2026.0527.27——
ScaleOTSetting=Emulator Zero-shot (Emu. ZS), Backbone=OPT-1.3B2025.0727.2——
Eso-LM (interpolating)Zero-shot=true, Likelihood-based evaluation=true, Number of Parameters=1.7B2026.0526.1——
ChanceZero-shot=true, Likelihood-based evaluation=true2026.0524.2——
OT†Setting=Emulator Zero-shot (Emu. ZS), Backbone=OPT-1.3B2025.0722.7——
GradOTSetting=Emulator Zero-shot (Emu. ZS), Backbone=OPT-1.3B2025.0722.3——
Gemma (2B)Size=2B2026.0214.32——