Synthetic Needle-In-A-Haystack retrieval on RULER S-NIAH (1K, 2K, 4K context)
100Accuracy (S-NIAH-1, 1K Context)Mamba2
Evaluation results
| Method | Links | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| Mamba2Model Parameters=1.3B2026.06 | 100 | 99.8 | 99 | 99.8 | 95.4 | 57 | 76 | 50.6 | 11.6 | 76.58 | |
| DeltaNetModel Parameters=1.3B2026.06 | 100 | 100 | 100 | 99.8 | 93.6 | 49.6 | 87.4 | 75.8 | 25.4 | 81.29 | |
| Gated DeltaNetModel Parameters=1.3B2026.06 | 100 | 100 | 100 | 100 | 99.8 | 76.6 | 83.8 | 70 | 21.4 | 83.51 | |
| Q-DeltaModel Parameters=1.3B2026.06 | 100 | 100 | 100 | 100 | 99.4 | 94.2 | 94.6 | 74 | 48 | 90.02 | |
| MambaModel Parameters=1.3B2026.06 | 99.8 | 99.6 | 87 | 98.8 | 92.8 | 50.8 | 22 | 12 | 0.8 | 62.62 | |
| RetNetModel Parameters=1.3B2026.06 | 96.6 | 27.8 | 7.4 | 99.4 | 60.8 | 24.4 | 20 | 5.2 | 1.2 | 38.09 |