Question Answering on ARC Challenge (Accuracy and Delta Accuracy)
20.99AccuracyAGoQ
Evaluation Results
| Method | Links | ||
|---|---|---|---|
| AGoQModel=LLaMA3.2-1B, Training Tokens=10B tokens, Evaluation Protocol=Zero-shot2026.05 | 20.99 | 0.0222 | |
| Megatron-LM FP16 BaselineModel=LLaMA2-7B, Training Tokens=2B tokens, Evaluation Protocol=Zero-shot2026.05 | 19.88 | — | |
| Megatron-LM FP16 BaselineModel=LLaMA3.2-1B, Training Tokens=10B tokens, Evaluation Protocol=Zero-shot2026.05 | 18.77 | — | |
| AGoQModel=LLaMA2-7B, Training Tokens=2B tokens, Evaluation Protocol=Zero-shot2026.05 | 18.34 | -0.0154 | |
| HRM-Text 1BArchitecture=Recurrent, FLOPs ($10^{21}$)=1, Tokens (T)=0.062026.05 | 0.819 | — | |
| Olmo3 7BArchitecture=Dense, FLOPs ($10^{21}$)=252, Tokens (T)=62026.05 | 0.816 | — | |
| Qwen3.5 2BArchitecture=Dense, FLOPs ($10^{21}$)=432, Tokens (T)=362026.05 | 0.81 | — | |
| Llama3.2 3BArchitecture=Dense, FLOPs ($10^{21}$)=162, Tokens (T)=92026.05 | 0.691 | — | |
| Ouro 1.4BArchitecture=Recurrent, FLOPs ($10^{21}$)=259, Tokens (T)=72026.05 | 0.609 | — | |
| OursQBackbone=Qwen3-MoE-30B-A3B, Type=P25% Q4b, Storage=11.64GB, Evaluation Protocol=zero-shot2026.06 | 0.5794 | — | |
| Gemma3 4BArchitecture=Dense, FLOPs ($10^{21}$)=96, Tokens (T)=42026.05 | 0.562 | — | |
| OriginalBase Model=LLaMA3-8B, Evaluation Strategy=Full model evaluation2026.05 | 0.5315 | — | |
| BaselineBackbone=Qwen3-MoE-30B-A3B, Type=–, Storage=61.06GB, Evaluation Protocol=zero-shot2026.06 | 0.527 | — | |
| OursBackbone=Qwen3-MoE-30B-A3B, Type=P50%, Storage=32.07GB, Evaluation Protocol=zero-shot2026.06 | 0.5213 | — | |
| PuzzleMoEBackbone=Qwen3-MoE-30B-A3B, Type=M25%, Storage=46.56GB, Evaluation Protocol=zero-shot2026.06 | 0.516 | — | |
| PuzzleMoEBackbone=Qwen3-MoE-30B-A3B, Type=M50%, Storage=32.07GB, Evaluation Protocol=zero-shot2026.06 | 0.51 | — | |
| DenseModel=LLaMA-2-7B, Sparsity Pattern=Dense (None), Evaluation Protocol=Zero-shot2026.06 | 0.4846 | — | |
| TaylorPruning Method=Taylor, Base Model=LLaMA3-8B, Calibration Dataset Size=1,500-instance, Evaluation Strategy=Task-independent structured pruning2026.05 | 0.4539 | — | |
| OursQBackbone=Qwen1.5-MoE-A2.7B, Type=P25% Q4b, Storage=5.60GB, Evaluation Protocol=zero-shot2026.06 | 0.4513 | — | |
| Out. Cosine-SimPruning Method=Out. Cosine-Sim, Base Model=LLaMA3-8B, Calibration Dataset Size=1,500-instance, Evaluation Strategy=Task-independent structured pruning2026.05 | 0.442 | — | |
| Out. Divergence-SimPruning Method=Out. Divergence-Sim, Base Model=LLaMA3-8B, Calibration Dataset Size=1,500-instance, Evaluation Strategy=Task-independent structured pruning2026.05 | 0.4354 | — | |
| Cosine SimilarityPruning Method=Cosine Similarity, Base Model=LLaMA3-8B, Calibration Dataset Size=1,500-instance, Evaluation Strategy=Task-independent structured pruning2026.05 | 0.4334 | — | |
| Out. Norm-SimPruning Method=Out. Norm-Sim, Base Model=LLaMA3-8B, Calibration Dataset Size=1,500-instance, Evaluation Strategy=Task-independent structured pruning2026.05 | 0.4266 | — | |
| CRePEModel=LLaMA-2-7B, Sparsity Pattern=unstructured 50%, Evaluation Protocol=Zero-shot2026.06 | 0.4215 | — | |
| MoNEBackbone=Qwen1.5-MoE-A2.7B, Type=P25%, Storage=22.40GB, Evaluation Protocol=zero-shot2026.06 | 0.4215 | — | |
| WandaBackbone=Qwen1.5-MoE-A2.7B, Type=P25% Q4b, Storage=7.10GB, Evaluation Protocol=zero-shot2026.06 | 0.4164 | — | |
| EAC-MoEBackbone=Qwen1.5-MoE-A2.7B, Type=P38% Q3.03b, Storage=4.35GB, Evaluation Protocol=zero-shot2026.06 | 0.413 | — | |
| MoE-I2Backbone=Qwen1.5-MoE-A2.7B, Type=L53.98%, Storage=15.18GB, Evaluation Protocol=zero-shot2026.06 | 0.4113 | — | |
| PuzzleMoEBackbone=Qwen1.5-MoE-A2.7B, Type=M25%, Storage=22.40GB, Evaluation Protocol=zero-shot2026.06 | 0.409 | — | |
| WandaBackbone=Qwen1.5-MoE-A2.7B, Type=P25%, Storage=28.67GB, Evaluation Protocol=zero-shot2026.06 | 0.407 | — | |
| PuzzleMoEBackbone=Qwen1.5-MoE-A2.7B, Type=M50%, Storage=16.17GB, Evaluation Protocol=zero-shot2026.06 | 0.407 | — | |
| OursBackbone=Qwen1.5-MoE-A2.7B, Type=P50%, Storage=16.17GB, Evaluation Protocol=zero-shot2026.06 | 0.4053 | — | |
| BaselineBackbone=Qwen1.5-MoE-A2.7B, Type=–, Storage=28.67GB, Evaluation Protocol=zero-shot2026.06 | 0.4041 | — | |
| MoNEBackbone=Qwen1.5-MoE-A2.7B, Type=P25% Q4b, Storage=5.60GB, Evaluation Protocol=zero-shot2026.06 | 0.4036 | — | |
| PerplexityPruning Method=Perplexity, Base Model=LLaMA3-8B, Calibration Dataset Size=1,500-instance, Evaluation Strategy=Task-independent structured pruning2026.05 | 0.4019 | — | |
| C-PruneBackbone=Qwen1.5-MoE-A2.7B, Type=P25% Q4b, Storage=5.60GB, Evaluation Protocol=zero-shot2026.06 | 0.4 | — | |
| He et al.Backbone=Qwen1.5-MoE-A2.7B, Type=P25% Q4b, Storage=5.60GB, Evaluation Protocol=zero-shot2026.06 | 0.3891 | — | |
| WandaBackbone=Qwen1.5-MoE-A2.7B, Type=P50%, Storage=28.67GB, Evaluation Protocol=zero-shot2026.06 | 0.3848 | — | |
| Huginn 3.5BArchitecture=Recurrent, FLOPs ($10^{21}$)=127, Tokens (T)=0.82026.05 | 0.382 | — | |
| Acc (Ours)Pruning Method=Accuracy-based relevance score, Base Model=LLaMA3-8B, Calibration Dataset Size=1,500-instance, Evaluation Strategy=Task-independent structured pruning2026.05 | 0.3669 | — | |
| CRePEModel=LLaMA-2-7B, Sparsity Pattern=2:4, Evaluation Protocol=Zero-shot2026.06 | 0.3652 | — | |
| Extra-MergeZero-shot=true, Backbone=Pythia-12B2026.05 | 0.3276 | — | |
| BLADETrain Toks.=3B, Backbone=TinyLlama-1.1B2026.06 | 0.3259 | — | |
| PMA (average)Zero-shot=true, Backbone=Pythia-12B2026.05 | 0.3221 | — | |
| MoNEBackbone=Qwen1.5-MoE-A2.7B, Type=P50%, Storage=16.17GB, Evaluation Protocol=zero-shot2026.06 | 0.3217 | — | |
| PMA (EMAα=0.1)Zero-shot=true, Backbone=Pythia-12B, α=0.12026.05 | 0.32 | — | |
| PMA (EMAα=0.2)Zero-shot=true, Backbone=Pythia-12B, α=0.22026.05 | 0.3191 | — | |
| Pythia-12B (Raw)Zero-shot=true, Backbone=Pythia-12B, Checkpoint=step 140k2026.05 | 0.3166 | — | |
| Base-CTTrain Toks.=5B, Backbone=TinyLlama-1.1B2026.06 | 0.3148 | — | |
| RHO-1Train Toks.=3B, Backbone=TinyLlama-1.1B2026.06 | 0.3123 | — | |
| RandomTrain Toks.=3B, Backbone=TinyLlama-1.1B2026.06 | 0.3106 | — | |
| MATES*Train Toks.=3B, Backbone=TinyLlama-1.1B2026.06 | 0.3106 | — | |
| BaseTrain Toks.=-, Backbone=TinyLlama-1.1B2026.06 | 0.3012 | — | |
| He et al.Backbone=Qwen1.5-MoE-A2.7B, Type=P50%, Storage=16.17GB, Evaluation Protocol=zero-shot2026.06 | 0.2696 | — | |
| WandaBackbone=Qwen3-MoE-30B-A3B, Type=P25%, Storage=61.06GB, Evaluation Protocol=zero-shot2026.06 | 0.2645 | — | |
| WandaBackbone=Qwen3-MoE-30B-A3B, Type=P50%, Storage=61.06GB, Evaluation Protocol=zero-shot2026.06 | 0.2457 | — |