Common Sense Reasoning on ARC-Challenge (0-shot)
57.2Accuracyfp16
Evaluation Results
| Method | Links | |
|---|---|---|
| fp16Size=70B2026.02 | 57.2 | |
| MagRSize=70B2026.02 | 56.2 | |
| AstroSize=70B2026.02 | 54.8 | |
| OmniQuantSize=70B2026.02 | 54.3 | |
| fp16Size=13B2026.02 | 48.8 | |
| MagRSize=13B2026.02 | 47.4 | |
| OmniQuantSize=13B2026.02 | 46.5 | |
| AstroSize=13B2026.02 | 46.2 | |
| Llama2-7BBackbone=Llama2-7B2026.02 | 45.5 | |
| fp16Size=7B2026.02 | 45.1 | |
| AstroSize=7B2026.02 | 43.3 | |
| OmniQuantSize=7B2026.02 | 42.2 | |
| MagRSize=7B2026.02 | 40.5 | |
| Gated DeltaNet-H1Setting=Zero-shot, Model Size=1.3B parameters2025.10 | 40.1 | |
| COMPOTBackbone=Llama2-7B, Training-free=true, CR=0.22026.02 | 39.6 | |
| Nirvana (Ours)Setting=Zero-shot, Model Size=1.3B parameters2025.10 | 39.51 | |
| Gated DeltaNet-H2Setting=Zero-shot, Model Size=1.3B parameters2025.10 | 39.07 | |
| Gated DeltaNetSetting=Zero-shot, Model Size=1.3B parameters2025.10 | 38.39 | |
| Mamba2Setting=Zero-shot, Model Size=1.3B parameters2025.10 | 37.88 | |
| Nirvana-noTriggerSetting=Zero-shot, Model Size=1.3B parameters2025.10 | 37.84 | |
| SambaSetting=Zero-shot, Model Size=1.3B parameters2025.10 | 36.17 | |
| DeltaNetSetting=Zero-shot, Model Size=1.3B parameters2025.10 | 35.66 | |
| MambaSetting=Zero-shot, Model Size=1.3B parameters2025.10 | 35.4 | |
| HGRN2Setting=Zero-shot, Model Size=1.3B parameters2025.10 | 35.32 | |
| Transformer++Setting=Zero-shot, Model Size=1.3B parameters2025.10 | 35.1 | |
| RetNetSetting=Zero-shot, Model Size=1.3B parameters2025.10 | 33.78 | |
| Dobi-SVDBackbone=Llama2-7B, Training-free=false, CR=0.22026.02 | 29.3 | |
| COMPOTBackbone=Llama2-7B, Training-free=true, CR=0.42026.02 | 28.5 | |
| Dobi-SVDBackbone=Llama2-7B, Training-free=false, CR=0.62026.02 | 24.7 | |
| Dobi-SVDBackbone=Llama2-7B, Training-free=false, CR=0.42026.02 | 24.5 | |
| COMPOTBackbone=Llama2-7B, Training-free=true, CR=0.62026.02 | 18.8 |