Multiple Choice Question Answering on ARC-Easy (test)
89.1AccuracyTaT
Evaluation Results
| Method | Links | |
|---|---|---|
| TaTTrain Dataset=ARC-E2026.03 | 89.1 | |
| TaTTrain Dataset=OpenQA2026.03 | 87.12 | |
| TaTTrain Dataset=ARC-C2026.03 | 85.31 | |
| Few-shot AccuracyMode=Few-shot2026.03 | 84.3 | |
| Linear ProbeTrain Dataset=ARC-E2026.03 | 83.99 | |
| TaTTrain Dataset=CosQA2026.03 | 83.59 | |
| Linear ProbeTrain Dataset=ARC-C2026.03 | 80.09 | |
| TaTTrain Dataset=ComQA2026.03 | 80.05 | |
| Zero-shot AccuracyMode=Zero-shot2026.03 | 78.5 | |
| Linear ProbeTrain Dataset=ComQA2026.03 | 76.27 | |
| Linear ProbeTrain Dataset=CosQA2026.03 | 75.27 | |
| TaTTrain Dataset=Hellaswag2026.03 | 74.92 | |
| TaTTrain Dataset=SiQA2026.03 | 71.72 | |
| SAES-SVDRatio=0.2, Model=LLaMA-13B, Fine-tuning=false, Mixed-rank strategies=false, Zero-shot evaluation=true2026.02 | 71.2 | |
| Pythia-12BParameters=12B, Zero-shot=true2023.04 | 70.2 | |
| Linear ProbeTrain Dataset=OpenQA2026.03 | 69.44 | |
| SVD-LLMRatio=0.2, Model=LLaMA-13B, Fine-tuning=true, Mixed-rank strategies=false, Zero-shot evaluation=true2026.02 | 68.3 | |
| Dip-SVDRatio=0.2, Model=LLaMA-13B, Fine-tuning=false, Mixed-rank strategies=true, Zero-shot evaluation=true2026.02 | 68.1 | |
| ClassActBackbone=OpenLLaMA 3B, Checkpoint Selection Metric=next token prediction accuracy2024.11 | 67.34 | |
| Pythia-6.9BParameters=6.9B, Zero-shot=true2023.04 | 67.3 | |
| BADSBackbone=OpenLLaMA 3B, Checkpoint Selection Metric=next token prediction accuracy2024.11 | 67 | |
| AskLLM-OBackbone=OpenLLaMA 3B, Checkpoint Selection Metric=next token prediction accuracy2024.11 | 66.88 | |
| Random_SelectBackbone=OpenLLaMA 3B, Checkpoint Selection Metric=next token prediction accuracy2024.11 | 66.75 | |
| Duplicate_MetaBackbone=OpenLLaMA 3B, Checkpoint Selection Metric=next token prediction accuracy2024.11 | 65.15 | |
| MixingBackbone=OpenLLaMA 3B, Checkpoint Selection Metric=next token prediction accuracy2024.11 | 64.65 | |
| Pythia-2.8BParameters=2.8B, Zero-shot=true2023.04 | 64.4 | |
| TaTTrain Dataset=BoolQ2026.03 | 62.75 | |
| Pythia-1.4BParameters=1.4B, Zero-shot=true2023.04 | 60.6 | |
| CDSBackbone=OpenLLaMA 3B, Checkpoint Selection Metric=next token prediction accuracy2024.11 | 60.14 | |
| Linear ProbeTrain Dataset=Hellaswag2026.03 | 58.6 | |
| Mistral (Full-Attention)Model Scale=1.4B2024.07 | 58.2 | |
| Pythia-1BParameters=1B, Zero-shot=true2023.04 | 56.9 | |
| Dip-SVDRatio=0.4, Model=LLaMA-13B, Fine-tuning=false, Mixed-rank strategies=true, Zero-shot evaluation=true2026.02 | 54.8 | |
| BMoJo (Fading + Eidetic)Model Scale=1.4B2024.07 | 54.7 | |
| Linear ProbeTrain Dataset=SiQA2026.03 | 54.53 | |
| SAES-SVDRatio=0.4, Model=LLaMA-13B, Fine-tuning=false, Mixed-rank strategies=false, Zero-shot evaluation=true2026.02 | 54.3 | |
| Mamba (SSM)Model Scale=1.4B2024.07 | 52.4 | |
| BMoJo (Fading)Model Scale=1.4B2024.07 | 52.3 | |
| Meta_OnlyBackbone=OpenLLaMA 3B, Checkpoint Selection Metric=next token prediction accuracy2024.11 | 52.23 | |
| Pythia-410MParameters=410M, Zero-shot=true2023.04 | 52.1 | |
| SVD-LLMRatio=0.4, Model=LLaMA-13B, Fine-tuning=true, Mixed-rank strategies=false, Zero-shot evaluation=true2026.02 | 52.1 | |
| Hybrid (Sliding Attention + SSM)Model Scale=1.4B2024.07 | 48.4 | |
| Linear ProbeTrain Dataset=BoolQ2026.03 | 46.61 | |
| Mamba (SSM)Model Scale=370M2024.07 | 45 | |
| AR Transformer2026.02 | 44.95 | |
| Mistral (Full-Attention)Model Scale=370M2024.07 | 44.9 | |
| BMoJo (Fading + Eidetic)Model Scale=370M2024.07 | 44.3 | |
| Pythia-160MParameters=160M, Zero-shot=true2023.04 | 43.5 | |
| BMoJo (Fading)Model Scale=370M2024.07 | 43.1 | |
| Hybrid (Sliding Attention + SSM)Model Scale=370M2024.07 | 42.7 | |
| Pythia-70MParameters=70M, Zero-shot=true2023.04 | 37.4 | |
| MDLM2026.02 | 34.26 | |
| Duo++k=32026.02 | 28.28 | |
| Duo2026.02 | 28.11 | |
| Duo++k=52026.02 | 28.03 | |
| Duo++k=22026.02 | 27.32 | |
| GenMC_T5Model Size=LARGE2022.04 | 0.6901 | |
| UnifiedQA_T5-FTModel Size=LARGE, Protocol=Fine-Tuning2022.04 | 0.6692 | |
| UnifiedQA_T5Model Size=LARGE2022.04 | 0.6643 | |
| UnifiedQA_T5*Model Size=LARGE, Reference=Khashabi et al. (2020)2022.04 | 0.659 | |
| RoBERTaModel Size=LARGE2022.04 | 0.624 | |
| GenMC_T5Model Size=BASE2022.04 | 0.5882 | |
| UnifiedQA_T5-FTModel Size=BASE, Protocol=Fine-Tuning2022.04 | 0.5388 | |
| ALBERTModel Size=LARGE2022.04 | 0.5377 | |
| UnifiedQA_T5Model Size=BASE2022.04 | 0.5366 | |
| RoBERTaModel Size=BASE2022.04 | 0.5232 | |
| ALBERTModel Size=BASE2022.04 | 0.4584 | |
| UnifiedQA_T5*Model Size=BASE, Reference=Khashabi et al. (2020)2022.04 | 0.424 |