Commonsense Reasoning on MMLU
78.95AccuracyIAP-mv
Evaluation Results
| Method | Links | |
|---|---|---|
| IAP-mvBackbone=Qwen-14B-Chat, Zero-shot CoT=true2024.09 | 78.95 | |
| VanillaBackbone=Qwen3-MoE-30B-A3B, Sparsity=0%2025.11 | 77.8 | |
| IAP-ssBackbone=Qwen-14B-Chat, Zero-shot CoT=true2024.09 | 77.07 | |
| Prompt #5Backbone=Qwen-14B-Chat, Zero-shot CoT=true2024.09 | 76.84 | |
| PuzzleMoEBackbone=Qwen3-MoE-30B-A3B, Sparsity=25%2025.11 | 76.6 | |
| PuzzleMoEBackbone=Qwen3-MoE-30B-A3B, Sparsity=50%2025.11 | 75.1 | |
| Prompt #4Backbone=Qwen-14B-Chat, Zero-shot CoT=true2024.09 | 74.21 | |
| WandaBackbone=Qwen3-MoE-30B-A3B, Sparsity=2:42025.11 | 72.1 | |
| Llama-3.1-8BArchitecture=Transformer, Tokens (B)=15000, Evaluation Protocol=5-shot2025.05 | 68 | |
| VanillaBackbone=Mixtral-8x7B-v0.1, Sparsity=0%2025.11 | 67.9 | |
| PuzzleMoEBackbone=Mixtral-8x7B-v0.1, Sparsity=25%2025.11 | 66.8 | |
| D2Backbone=Qwen3-MoE-30B-A3B, Sparsity=20%2025.11 | 66.2 | |
| PuzzleMoEBackbone=Mixtral-8x7B-v0.1, Sparsity=50%2025.11 | 65.7 | |
| Falcon3-Mamba-7BArchitecture=Subquadratic (Pretrained), Tokens (B)=7300, Evaluation Protocol=5-shot2025.05 | 65 | |
| Sub-MoEBackbone=Qwen3-MoE-30B-A3B, Sparsity=25%2025.11 | 65 | |
| AMV (#1-7)Backbone=Qwen-14B-Chat, Zero-shot CoT=true2024.09 | 63.86 | |
| STUNBackbone=Mixtral-8x7B-v0.1, Sparsity=25%2025.11 | 63.3 | |
| WandaBackbone=Mixtral-8x7B-v0.1, Sparsity=2:42025.11 | 62 | |
| Prompt #2Backbone=Qwen-14B-Chat, Zero-shot CoT=true2024.09 | 61.93 | |
| VanillaBackbone=Qwen1.5-MoE-A2.7B, Sparsity=0%2025.11 | 61 | |
| HC-SMoEBackbone=Qwen3-MoE-30B-A3B, Sparsity=25%2025.11 | 60.8 | |
| PuzzleMoEBackbone=Qwen1.5-MoE-A2.7B, Sparsity=25%2025.11 | 60.4 | |
| Llamba-8BArchitecture=Subquadratic (Distilled), Tokens (B)=12, Evaluation Protocol=5-shot2025.05 | 60 | |
| PuzzleMoEBackbone=Qwen1.5-MoE-A2.7B, Sparsity=50%2025.11 | 60 | |
| IAP-mvBackbone=LLaMA-3-8B-Instruct, Zero-shot CoT=true2024.09 | 59.65 | |
| HC-SMoEBackbone=Mixtral-8x7B-v0.1, Sparsity=25%2025.11 | 59.4 | |
| Sub-MoEBackbone=Mixtral-8x7B-v0.1, Sparsity=25%2025.11 | 59 | |
| NAEEBackbone=Mixtral-8x7B-v0.1, Sparsity=25%2025.11 | 58.1 | |
| LoLA-8B (ours)Architecture=Subquadratic (Distilled), Tokens (B)=0.04, Evaluation Protocol=5-shot, η=64, λ=642025.05 | 57.6 | |
| Prompt #7Backbone=Qwen-14B-Chat, Zero-shot CoT=true2024.09 | 57.19 | |
| Prompt #6Backbone=LLaMA-3-8B-Instruct, Zero-shot CoT=true2024.09 | 56.67 | |
| IAP-ssBackbone=LLaMA-3-8B-Instruct, Zero-shot CoT=true2024.09 | 56.49 | |
| Sub-MoEBackbone=Qwen3-MoE-30B-A3B, Sparsity=50%2025.11 | 56 | |
| WandaBackbone=Qwen1.5-MoE-A2.7B, Sparsity=2:42025.11 | 55.8 | |
| Prompt #1Backbone=LLaMA-3-8B-Instruct, Zero-shot CoT=true2024.09 | 55.79 | |
| Prompt #6Backbone=Qwen-14B-Chat, Zero-shot CoT=true2024.09 | 55.25 | |
| RecurrentGemma-9BArchitecture=Subquadratic (Pretrained), Tokens (B)=2000, Evaluation Protocol=5-shot2025.05 | 55.1 | |
| LoLCATs-8BArchitecture=Subquadratic (Distilled), Tokens (B)=0.04, Evaluation Protocol=5-shot2025.05 | 54.9 | |
| IAP-mvBackbone=LLaMA-3 70B, Setting=Zero-shot CoT2024.09 | 54.39 | |
| Self-discoverBackbone=LLaMA-3-8B-Instruct, Zero-shot CoT=true2024.09 | 52.63 | |
| Prompt #5Backbone=LLaMA-3 70B, Setting=Zero-shot CoT2024.09 | 52.63 | |
| AMV (all)Backbone=LLaMA-3-8B-Instruct, Zero-shot CoT=true2024.09 | 52.53 | |
| AMV (all)Backbone=Qwen-14B-Chat, Zero-shot CoT=true2024.09 | 52.46 | |
| D2Backbone=Qwen1.5-MoE-A2.7B, Sparsity=20%2025.11 | 52.4 | |
| Prompt #4Backbone=LLaMA-3 70B, Setting=Zero-shot CoT2024.09 | 52.11 | |
| HC-SMoEBackbone=Qwen1.5-MoE-A2.7B, Sparsity=25%2025.11 | 51 | |
| Prompt #2Backbone=LLaMA-3-8B-Instruct, Zero-shot CoT=true2024.09 | 50.35 | |
| Prompt #2Backbone=LLaMA-3 70B, Setting=Zero-shot CoT2024.09 | 50.18 | |
| HC-SMoEBackbone=Mixtral-8x7B-v0.1, Sparsity=50%2025.11 | 49 | |
| Prompt #4Backbone=LLaMA-3-8B-Instruct, Zero-shot CoT=true2024.09 | 48.95 | |
| Mamba2-8BArchitecture=Subquadratic (Pretrained), Tokens (B)=3500, Evaluation Protocol=5-shot2025.05 | 48.7 | |
| Prompt #3Backbone=Qwen-14B-Chat, Zero-shot CoT=true2024.09 | 48.25 | |
| Sub-MoEBackbone=Mixtral-8x7B-v0.1, Sparsity=50%2025.11 | 48 | |
| D2Backbone=Qwen3-MoE-30B-A3B, Sparsity=40%2025.11 | 47.9 | |
| NAEEBackbone=Mixtral-8x7B-v0.1, Sparsity=50%2025.11 | 47.3 | |
| Prompt #5Backbone=LLaMA-3-8B-Instruct, Zero-shot CoT=true2024.09 | 46.67 | |
| Prompt #3Backbone=LLaMA-3 70B, Setting=Zero-shot CoT2024.09 | 44.56 | |
| Prompt #3Backbone=LLaMA-3-8B-Instruct, Zero-shot CoT=true2024.09 | 43.51 | |
| Prompt #9Backbone=LLaMA-3 70B, Setting=Zero-shot CoT2024.09 | 43.33 | |
| Mamba2-Llama3-8B (L3.1-Instr.)Architecture=Subquadratic (Distilled), Tokens (B)=20, Evaluation Protocol=5-shot2025.05 | 43.2 | |
| Prompt #1Backbone=Qwen-14B-Chat, Zero-shot CoT=true2024.09 | 42.48 | |
| Prompt #7Backbone=LLaMA-3-8B-Instruct, Zero-shot CoT=true2024.09 | 42.28 | |
| AMV (#1-7)Backbone=LLaMA-3-8B-Instruct, Zero-shot CoT=true2024.09 | 41.23 | |
| HC-SMoEBackbone=Qwen3-MoE-30B-A3B, Sparsity=50%2025.11 | 39.6 | |
| Griffin 7BArchitecture=Subquadratic (Pretrained), Tokens (B)=300, Evaluation Protocol=5-shot2025.05 | 39.3 | |
| VanillaBackbone=Deepseek-MoE-16b, Sparsity=0%2025.11 | 37.8 | |
| D2Backbone=Qwen1.5-MoE-A2.7B, Sparsity=40%2025.11 | 37.5 | |
| In-Place TTTArchitecture=Full Attention, Parameter Count=4B2026.04 | 37.42 | |
| PuzzleMoEBackbone=Deepseek-MoE-16b, Sparsity=25%2025.11 | 37.2 | |
| Prompt #1Backbone=LLaMA-3 70B, Setting=Zero-shot CoT2024.09 | 37.02 | |
| PuzzleMoEBackbone=Deepseek-MoE-16b, Sparsity=50%2025.11 | 36.9 | |
| In-Place TTTArchitecture=Sliding-Window Attention (SWA), Parameter Count=4B2026.04 | 36.48 | |
| Full AttentionModel Group=Baselines, Parameter Count=4B2026.04 | 36.43 | |
| Sliding-Window Attention (SWA)Model Group=Baselines, Parameter Count=4B2026.04 | 36.06 | |
| Prompt #6Backbone=LLaMA-3 70B, Setting=Zero-shot CoT2024.09 | 35.79 | |
| Hawk 7BArchitecture=Subquadratic (Pretrained), Tokens (B)=300, Evaluation Protocol=5-shot2025.05 | 35 | |
| WandaBackbone=Deepseek-MoE-16b, Sparsity=2:42025.11 | 31 | |
| Prompt #9Backbone=Qwen-14B-Chat, Zero-shot CoT=true2024.09 | 30.7 | |
| Mamba-8BArchitecture=Subquadratic (Pretrained), Tokens (B)=1100, Evaluation Protocol=5-shot2025.05 | 28 | |
| Sub-MoEBackbone=Deepseek-MoE-16b, Sparsity=25%2025.11 | 27 | |
| Prompt #8Backbone=LLaMA-3-8B-Instruct, Zero-shot CoT=true2024.09 | 26.32 | |
| HC-SMoEBackbone=Deepseek-MoE-16b, Sparsity=25%2025.11 | 24.5 | |
| Hedgehog-8B (Llama-3)Architecture=Subquadratic (Distilled), Tokens (B)=0.04, Evaluation Protocol=5-shot2025.05 | 24.2 | |
| HC-SMoEBackbone=Qwen1.5-MoE-A2.7B, Sparsity=50%2025.11 | 23.2 | |
| HC-SMoEBackbone=Deepseek-MoE-16b, Sparsity=50%2025.11 | 23 | |
| Prompt #8Backbone=LLaMA-3 70B, Setting=Zero-shot CoT2024.09 | 22.56 | |
| Sub-MoEBackbone=Deepseek-MoE-16b, Sparsity=50%2025.11 | 22 | |
| Prompt #9Backbone=LLaMA-3-8B-Instruct, Zero-shot CoT=true2024.09 | 20.18 | |
| Prompt #7Backbone=LLaMA-3 70B, Setting=Zero-shot CoT2024.09 | 14.56 | |
| Prompt #8Backbone=Qwen-14B-Chat, Zero-shot CoT=true2024.09 | 9.3 |