Common Sense Reasoning on ARC-Easy (0-shot)
79.5Accuracyfp16
Evaluation Results
| Method | Links | |
|---|---|---|
| fp16Size=70B2026.02 | 79.5 | |
| MagRSize=70B2026.02 | 79.5 | |
| AstroSize=70B2026.02 | 79.3 | |
| OmniQuantSize=70B2026.02 | 78.5 | |
| fp16Size=13B2026.02 | 76.6 | |
| MagRSize=13B2026.02 | 75.6 | |
| AstroSize=13B2026.02 | 74.7 | |
| fp16Size=7B2026.02 | 73.8 | |
| OmniQuantSize=13B2026.02 | 73.4 | |
| Mamba2Setting=Zero-shot, Model Size=1.3B parameters2025.10 | 72.47 | |
| Gated DeltaNet-H1Setting=Zero-shot, Model Size=1.3B parameters2025.10 | 71.75 | |
| Gated DeltaNet-H2Setting=Zero-shot, Model Size=1.3B parameters2025.10 | 71.33 | |
| Gated DeltaNetSetting=Zero-shot, Model Size=1.3B parameters2025.10 | 71.21 | |
| AstroSize=7B2026.02 | 71.2 | |
| OmniQuantSize=7B2026.02 | 71.1 | |
| TequilaLLM-3B#Tokens=10B, Quantization=W1.58A162026.06 | 70.2 | |
| Nirvana (Ours)Setting=Zero-shot, Model Size=1.3B parameters2025.10 | 69.92 | |
| MagRSize=7B2026.02 | 69.7 | |
| Qwen3-8B + CAT-Q#Tokens=1M, Quantization=W1.58A162026.06 | 69.65 | |
| MambaSetting=Zero-shot, Model Size=1.3B parameters2025.10 | 69.52 | |
| HGRN2Setting=Zero-shot, Model Size=1.3B parameters2025.10 | 69.4 | |
| Transformer++Setting=Zero-shot, Model Size=1.3B parameters2025.10 | 68.83 | |
| SambaSetting=Zero-shot, Model Size=1.3B parameters2025.10 | 68.81 | |
| Nirvana-noTriggerSetting=Zero-shot, Model Size=1.3B parameters2025.10 | 68.8 | |
| DeltaNetSetting=Zero-shot, Model Size=1.3B parameters2025.10 | 68.47 | |
| RetNetSetting=Zero-shot, Model Size=1.3B parameters2025.10 | 67.34 | |
| Qwen3-4B + CAT-Q#Tokens=2M, Quantization=W1.58A162026.06 | 66.41 | |
| TriLM-3.9B#Tokens=300B, Quantization=W1.58A162026.06 | 66 | |
| BitNetV1-3.9B#Tokens=100B, Quantization=W1.58A162026.06 | 64.2 | |
| Qwen3-4B + CAT-Q#Tokens=1M, Quantization=W1.58A162026.06 | 62.29 | |
| TernaryLLM-8B+KD#Tokens=1T, Quantization=W1.58A162026.06 | 61.2 | |
| BitNetV1-7B#Tokens=100B, Quantization=W1.58A162026.06 | 59.51 | |
| TriLM-1.5B#Tokens=300B, Quantization=W1.58A162026.06 | 59 | |
| Original EndpointModel=1B, Zero-shot evaluation=true2026.06 | 56.73 | |
| Qwen3-1.7B + CAT-Q#Tokens=1M, Quantization=W1.58A162026.06 | 55.88 | |
| BitNetV1-1.3B#Tokens=100B, Quantization=W1.58A162026.06 | 54.9 | |
| Original PythiaModel=410M, Zero-shot evaluation=true2026.06 | 51.89 | |
| Dual Learned MatchingModel=1B, Zero-shot evaluation=true2026.06 | 37.16 | |
| Learned MatchingModel=1B, Zero-shot evaluation=true2026.06 | 36.57 | |
| Dual Learned MatchingModel=410M, Zero-shot evaluation=true2026.06 | 35.69 | |
| Learned MatchingModel=410M, Zero-shot evaluation=true2026.06 | 33.25 | |
| Raw InterpolationModel=410M, Zero-shot evaluation=true2026.06 | 29.71 | |
| Raw InterpolationModel=1B, Zero-shot evaluation=true2026.06 | 29.08 |