List your company
ResearchFormerly WizWand

Multiple Choice Question Answering on ARC-Easy (test)

89.1AccuracyTaT
-3.1230420.8194844.76268.70452Apr 30, 2022Dec 19, 2022Aug 10, 2023Mar 30, 2024Nov 19, 2024Jul 10, 2025Mar 1, 2026
Updated 8mo ago

Evaluation results

MethodLinks
TaTTrain Dataset=ARC-E2026.0389.1
TaTTrain Dataset=OpenQA2026.0387.12
TaTTrain Dataset=ARC-C2026.0385.31
Few-shot AccuracyMode=Few-shot2026.0384.3
Linear ProbeTrain Dataset=ARC-E2026.0383.99
TaTTrain Dataset=CosQA2026.0383.59
Linear ProbeTrain Dataset=ARC-C2026.0380.09
TaTTrain Dataset=ComQA2026.0380.05
Zero-shot AccuracyMode=Zero-shot2026.0378.5
Linear ProbeTrain Dataset=ComQA2026.0376.27
Linear ProbeTrain Dataset=CosQA2026.0375.27
TaTTrain Dataset=Hellaswag2026.0374.92
TaTTrain Dataset=SiQA2026.0371.72
SAES-SVDRatio=0.2, Model=LLaMA-13B, Fine-tuning=false, Mixed-rank strategies=false, Zero-shot evaluation=true2026.0271.2
Pythia-12BParameters=12B, Zero-shot=true2023.0470.2
Linear ProbeTrain Dataset=OpenQA2026.0369.44
SVD-LLMRatio=0.2, Model=LLaMA-13B, Fine-tuning=true, Mixed-rank strategies=false, Zero-shot evaluation=true2026.0268.3
Dip-SVDRatio=0.2, Model=LLaMA-13B, Fine-tuning=false, Mixed-rank strategies=true, Zero-shot evaluation=true2026.0268.1
ClassActBackbone=OpenLLaMA 3B, Checkpoint Selection Metric=next token prediction accuracy2024.1167.34
Pythia-6.9BParameters=6.9B, Zero-shot=true2023.0467.3
BADSBackbone=OpenLLaMA 3B, Checkpoint Selection Metric=next token prediction accuracy2024.1167
AskLLM-OBackbone=OpenLLaMA 3B, Checkpoint Selection Metric=next token prediction accuracy2024.1166.88
Random_SelectBackbone=OpenLLaMA 3B, Checkpoint Selection Metric=next token prediction accuracy2024.1166.75
Duplicate_MetaBackbone=OpenLLaMA 3B, Checkpoint Selection Metric=next token prediction accuracy2024.1165.15
MixingBackbone=OpenLLaMA 3B, Checkpoint Selection Metric=next token prediction accuracy2024.1164.65
Pythia-2.8BParameters=2.8B, Zero-shot=true2023.0464.4
TaTTrain Dataset=BoolQ2026.0362.75
Pythia-1.4BParameters=1.4B, Zero-shot=true2023.0460.6
CDSBackbone=OpenLLaMA 3B, Checkpoint Selection Metric=next token prediction accuracy2024.1160.14
Linear ProbeTrain Dataset=Hellaswag2026.0358.6
Mistral (Full-Attention)Model Scale=1.4B2024.0758.2
Pythia-1BParameters=1B, Zero-shot=true2023.0456.9
Dip-SVDRatio=0.4, Model=LLaMA-13B, Fine-tuning=false, Mixed-rank strategies=true, Zero-shot evaluation=true2026.0254.8
BMoJo (Fading + Eidetic)Model Scale=1.4B2024.0754.7
Linear ProbeTrain Dataset=SiQA2026.0354.53
SAES-SVDRatio=0.4, Model=LLaMA-13B, Fine-tuning=false, Mixed-rank strategies=false, Zero-shot evaluation=true2026.0254.3
Mamba (SSM)Model Scale=1.4B2024.0752.4
BMoJo (Fading)Model Scale=1.4B2024.0752.3
Meta_OnlyBackbone=OpenLLaMA 3B, Checkpoint Selection Metric=next token prediction accuracy2024.1152.23
Pythia-410MParameters=410M, Zero-shot=true2023.0452.1
SVD-LLMRatio=0.4, Model=LLaMA-13B, Fine-tuning=true, Mixed-rank strategies=false, Zero-shot evaluation=true2026.0252.1
Hybrid (Sliding Attention + SSM)Model Scale=1.4B2024.0748.4
Linear ProbeTrain Dataset=BoolQ2026.0346.61
Mamba (SSM)Model Scale=370M2024.0745
AR Transformer2026.0244.95
Mistral (Full-Attention)Model Scale=370M2024.0744.9
BMoJo (Fading + Eidetic)Model Scale=370M2024.0744.3
Pythia-160MParameters=160M, Zero-shot=true2023.0443.5
BMoJo (Fading)Model Scale=370M2024.0743.1
Hybrid (Sliding Attention + SSM)Model Scale=370M2024.0742.7
Pythia-70MParameters=70M, Zero-shot=true2023.0437.4
MDLM2026.0234.26
Duo++k=32026.0228.28
Duo2026.0228.11
Duo++k=52026.0228.03
Duo++k=22026.0227.32
GenMC_T5Model Size=LARGE2022.040.6901
UnifiedQA_T5-FTModel Size=LARGE, Protocol=Fine-Tuning2022.040.6692
UnifiedQA_T5Model Size=LARGE2022.040.6643
UnifiedQA_T5*Model Size=LARGE, Reference=Khashabi et al. (2020)2022.040.659
RoBERTaModel Size=LARGE2022.040.624
GenMC_T5Model Size=BASE2022.040.5882
UnifiedQA_T5-FTModel Size=BASE, Protocol=Fine-Tuning2022.040.5388
ALBERTModel Size=LARGE2022.040.5377
UnifiedQA_T5Model Size=BASE2022.040.5366
RoBERTaModel Size=BASE2022.040.5232
ALBERTModel Size=BASE2022.040.4584
UnifiedQA_T5*Model Size=BASE, Reference=Khashabi et al. (2020)2022.040.424