List your company
ResearchFormerly WizWand

Multiple Choice Question Answering on ARC Challenge (test)

84.1AccuracyPPT (Distribution)
18.642435.636252.6369.6238Apr 30, 2022Jan 5, 2023Sep 13, 2023May 20, 2024Jan 26, 2025Oct 3, 2025Jun 11, 2026
Updated 4mo ago

Evaluation results

MethodLinks
PPT (Distribution)Backbone=QWEN2-7B2026.0584.1
PPT (Mean)Backbone=QWEN2-7B2026.0583.7
BaseBackbone=QWEN2-7B2026.0583.6
TaTTrain Dataset=ARC-C2026.0382.17
PPT (Distribution)Backbone=LLAMA3-8B2026.0579.6
BaseBackbone=LLAMA3-8B2026.0578.6
TaTTrain Dataset=OpenQA2026.0378.41
PPT (Mean)Backbone=LLAMA3-8B2026.0577.4
Linear ProbeTrain Dataset=ARC-E2026.0375.55
Linear ProbeTrain Dataset=ARC-C2026.0375.32
TaTTrain Dataset=ARC-E2026.0373.81
Linear ProbeTrain Dataset=ComQA2026.0371.72
Linear ProbeTrain Dataset=CosQA2026.0370.09
TaTTrain Dataset=CosQA2026.0369.71
Linear ProbeTrain Dataset=OpenQA2026.0366.42
TaTTrain Dataset=Hellaswag2026.0365.96
Few-shot AccuracyMode=Few-shot2026.0365.3
TaTTrain Dataset=ComQA2026.0360.92
TaTTrain Dataset=SiQA2026.0359.9
TWLABackbone=Qwen3-32B, Weight bits=1.58, Activation bits=16, Setting=Zero-shot2026.0655.33
VerbalizedBackbone=LLAMA3-8B2026.0554.8
UnifiedQA_T5-FTModel Size=LARGE, Protocol=Fine-Tuning2022.0454.42
UnifiedQA_T5*Model Size=LARGE, Reference=Khashabi et al. (2020)2022.0454.4
UnifiedQA_T5Model Size=LARGE2022.0454.33
TWLABackbone=LLaMA2-70B, Weight bits=1.58, Activation bits=16, Setting=Zero-shot2026.0653.75
TaTTrain Dataset=BoolQ2026.0353.5
Linear ProbeTrain Dataset=Hellaswag2026.0352.92
TWLABackbone=Qwen3-14B, Weight bits=1.58, Activation bits=16, Setting=Zero-shot2026.0652.3
Zero-shot AccuracyMode=Zero-shot2026.0350.1
Linear ProbeTrain Dataset=SiQA2026.0349.82
UNPRUNEDACT.=2.8B, MEM.=100%, Speedup=1.0x, SFT Protocol=w/o SFT2024.1148.3
GenMC_T5Model Size=LARGE2022.0447.41
TWLABackbone=Qwen3-8B, Weight bits=1.58, Activation bits=16, Setting=Zero-shot2026.0645.99
Linear ProbeTrain Dataset=BoolQ2026.0344.51
UnifiedQA_T5Model Size=BASE2022.0442.58
UnifiedQA_T5-FTModel Size=BASE, Protocol=Fine-Tuning2022.0442.43
TWLABackbone=LLaMA2-13B, Weight bits=1.58, Activation bits=16, Setting=Zero-shot2026.0642.41
CD-MoE-SR#L=9, ACT.=2.8B, MEM.=72.5%, Speedup=1.26x, SFT Protocol=w/ lightweight SFT2024.1142.4
CD-MoE-S#L=8, ACT.=2.4B, MEM.=73.1%, Speedup=1.34x, SFT Protocol=w/ lightweight SFT2024.1141.5
TWLABackbone=LLaMA3-8B, Weight bits=1.58, Activation bits=16, Setting=Zero-shot2026.0639.42
GenMC_T5Model Size=BASE2022.0439
CD-MoE-SR#L=9, ACT.=2.8B, MEM.=72.5%, Speedup=1.26x, SFT Protocol=w/o SFT2024.1137.9
CD-MoE-S#L=8, ACT.=2.4B, MEM.=73.1%, Speedup=1.34x, SFT Protocol=w/o SFT2024.1137.8
LayerDrop#L=8, ACT.=2.4B, MEM.=72.2%, Speedup=1.34x, SFT Protocol=w/ lightweight SFT2024.1137.8
TWLABackbone=LLaMA2-7B, Weight bits=1.58, Activation bits=16, Setting=Zero-shot2026.0637.63
M-SMoE#L=9, ACT.=2.8B, MEM.=70.0%, Speedup=1.0x, SFT Protocol=w/o SFT2024.1137.1
BlockDrop#L=8, ACT.=2.3B, MEM.=71.3%, Speedup=1.42x, SFT Protocol=w/o SFT2024.1136.2
OPENLLAMA-3BACT.=3B, SFT Protocol=w/o SFT2024.1136.1
RoBERTaModel Size=LARGE2022.0435.97
LayerDrop#L=8, ACT.=2.4B, MEM.=72.2%, Speedup=1.34x, SFT Protocol=w/o SFT2024.1135.3
AskLLM-OBackbone=OpenLLaMA 3B, Checkpoint Selection Metric=next token prediction accuracy2024.1135.15
RoBERTaModel Size=BASE2022.0434.85
BADSBackbone=OpenLLaMA 3B, Checkpoint Selection Metric=next token prediction accuracy2024.1134.39
MixingBackbone=OpenLLaMA 3B, Checkpoint Selection Metric=next token prediction accuracy2024.1133.79
Duplicate_MetaBackbone=OpenLLaMA 3B, Checkpoint Selection Metric=next token prediction accuracy2024.1133.28
Meta_OnlyBackbone=OpenLLaMA 3B, Checkpoint Selection Metric=next token prediction accuracy2024.1132.08
ClassActBackbone=OpenLLaMA 3B, Checkpoint Selection Metric=next token prediction accuracy2024.1131.91
OPT-2.7BACT.=2.7B, SFT Protocol=w/o SFT2024.1131.2
ALBERTModel Size=LARGE2022.0431.19
BLOOM-3BACT.=3B, SFT Protocol=w/o SFT2024.1130.4
ALBERTModel Size=BASE2022.0430.21
GPT-NEO-2.7BACT.=2.7B, SFT Protocol=w/o SFT2024.1129.8
Random_SelectBackbone=OpenLLaMA 3B, Checkpoint Selection Metric=next token prediction accuracy2024.1128.92
Duo++k=22026.0226.11
UnifiedQA_T5*Model Size=BASE, Reference=Khashabi et al. (2020)2022.0425.8
Duo++k=52026.0225.77
Duo2026.0225.43
Duo++k=32026.0225
MDLM2026.0224.66
AR Transformer2026.0223.04
CDSBackbone=OpenLLaMA 3B, Checkpoint Selection Metric=next token prediction accuracy2024.1121.16