Single needle-in-a-haystack retrieval on RULER S-NIAH
100S-NIAH PK Success Rate (2K)Transformer++
Evaluation results
| Method | Links | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| Transformer++Model Scale=1.3B, Training Context Length=4K2025.10 | 100 | 100 | 62.6 | 100 | 100 | 59.4 | 100 | 100 | 98.6 | 91.2 | |
| NirvanaModel Scale=1.3B, Training Context Length=4K2025.10 | 100 | 100 | 100 | 100 | 100 | 99.6 | 98.8 | 97.8 | 95.4 | 99.1 | |
| Nirvana-noTriggerModel Scale=1.3B, Training Context Length=4K2025.10 | 99.6 | 99.6 | 99 | 99.8 | 99.8 | 98.8 | 99 | 97.4 | 94.8 | 98.6 | |
| Gated DeltaNet-H2Model Scale=1.3B, Training Context Length=4K2025.10 | 99.2 | 97.8 | 97.4 | 98 | 97.8 | 96.2 | 98 | 97.4 | 96.8 | 97.6 | |
| SambaModel Scale=1.3B, Training Context Length=4K2025.10 | 98.8 | 98 | 97.4 | 98.8 | 98.6 | 96.2 | 97.4 | 96.8 | 90 | 96.9 | |
| Mamba2Model Scale=1.3B, Training Context Length=4K2025.10 | 98.6 | 61.4 | 31 | 98.4 | 55.8 | 14.2 | 62.2 | 42.2 | 4.2 | 52 | |
| TTTModel Scale=1.3B, Training Context Length=4K2025.10 | 98.4 | 98.8 | 98 | 60.2 | 36.6 | 10.2 | 85.8 | 78.8 | 28 | 66.1 | |
| DeltaNetModel Scale=1.3B, Training Context Length=4K2025.10 | 96.8 | 98.8 | 98.6 | 47.2 | 15.4 | 12.8 | 85.2 | 46.2 | 20 | 57.9 | |
| Gated DeltaNetModel Scale=1.3B, Training Context Length=4K2025.10 | 89.8 | 91.4 | 90 | 99.2 | 91.8 | 26.4 | 86.4 | 82.6 | 24.4 | 75.8 |