Commonsense Reasoning on LM-Evaluation-Harness Zero-Shot Suite
69.87ARC-Easy AccuracyMamba2
Evaluation results
| Method | Links | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| Mamba2number of parameters=1.3B, training tokens=30B, evaluation mode=zero-shot2026.06 | 69.87 | 36.35 | 53.24 | 40.68 | 70.29 | 56.04 | 55.81 | 37.4 | 52.46 | |
| Gated DeltaNetnumber of parameters=1.3B, training tokens=30B, evaluation mode=zero-shot2026.06 | 68.6 | 33.28 | 52.6 | 43.8 | 70.84 | 54.78 | 59.42 | 38.8 | 52.77 | |
| Q-Deltanumber of parameters=1.3B, training tokens=30B, evaluation mode=zero-shot2026.06 | 68.27 | 36.6 | 53.46 | 43.28 | 71.44 | 54.93 | 61.41 | 38.4 | 53.47 | |
| Mambanumber of parameters=1.3B, training tokens=30B, evaluation mode=zero-shot2026.06 | 68.1 | 36.18 | 53.44 | 40.77 | 72.2 | 55.01 | 55.63 | 37.8 | 52.39 | |
| DeltaNetnumber of parameters=1.3B, training tokens=30B, evaluation mode=zero-shot2026.06 | 67.63 | 34.47 | 51.09 | 41.78 | 70.95 | 54.7 | 61.19 | 38.4 | 52.53 | |
| RetNetnumber of parameters=1.3B, training tokens=30B, evaluation mode=zero-shot2026.06 | 63.68 | 33.36 | 47.73 | 38.7 | 69.04 | 52.72 | 60.61 | 36.6 | 50.31 | |
| Gated DeltaNetnumber of parameters=340M, training tokens=15B, evaluation mode=zero-shot2026.06 | 60.02 | 25.94 | 40.25 | 31.52 | 67.3 | 51.54 | 57.13 | 34.4 | 46.01 | |
| Mambanumber of parameters=340M, training tokens=15B, evaluation mode=zero-shot2026.06 | 59.68 | 29.18 | 42.98 | 32.97 | 67.52 | 51.78 | 55.81 | 33.2 | 46.64 | |
| Q-Deltanumber of parameters=340M, training tokens=15B, evaluation mode=zero-shot2026.06 | 59.51 | 28.5 | 41.61 | 33.9 | 67.63 | 52.88 | 59.48 | 34.4 | 47.24 | |
| Mamba2number of parameters=340M, training tokens=15B, evaluation mode=zero-shot2026.06 | 59.3 | 29.01 | 42.13 | 33.67 | 68.01 | 52.33 | 51.9 | 33.8 | 46.27 | |
| RetNetnumber of parameters=340M, training tokens=15B, evaluation mode=zero-shot2026.06 | 57.07 | 28.41 | 38.71 | 27.36 | 66.54 | 49.41 | 56.88 | 32 | 44.55 | |
| DeltaNetnumber of parameters=340M, training tokens=15B, evaluation mode=zero-shot2026.06 | 54.88 | 27.47 | 38.65 | 27.11 | 63.6 | 49.96 | 59.46 | 29.4 | 43.82 |