Needle-in-a-Haystack Retrieval on RULER
100S-NIAH-1 (Pass-Key Retrieval)Full attention
Evaluation results
| Method | Links | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| Full attentionContext Length=8K, Model Size=3B, Training Context Length=128K2025.12 | 100 | 99 | 64 | — | — | — | — | — | — | |
| Full attentionContext Length=16K, Model Size=3B, Training Context Length=128K2025.12 | 100 | 100 | 64 | — | — | — | — | — | — | |
| Full attentionContext Length=32K, Model Size=3B, Training Context Length=128K2025.12 | 100 | 100 | 67 | — | — | — | — | — | — | |
| Full attentionContext Length=64K, Model Size=3B, Training Context Length=128K2025.12 | 100 | 100 | 83 | — | — | — | — | — | — | |
| SWAContext Length=8K, Model Size=3B, Training Context Length=128K2025.12 | 100 | 100 | 57 | — | — | — | — | — | — | |
| Hybrid SWA and fullContext Length=8K, Model Size=3B, Training Context Length=128K2025.12 | 100 | 100 | 63 | — | — | — | — | — | — | |
| Gated DeltaNetContext Length=8K, Model Size=3B, Training Context Length=128K2025.12 | 100 | 100 | 91 | — | — | — | — | — | — | |
| TTT-E2EContext Length=8K, Model Size=3B, Training Context Length=128K2025.12 | 100 | 99 | 77 | — | — | — | — | — | — | |
| RAT+dilation size (D)=1, context length (T)=4096, Training Mode=SFT2026.02 | 100 | — | 100 | 100 | 100 | 99.6 | 99.85 | 100 | — | |
| RAT+dilation size (D)=2, context length (T)=4096, Training Mode=SFT2026.02 | 100 | — | 99.2 | 100 | 99.2 | 96 | 99.8 | 99.8 | — | |
| RAT+dilation size (D)=4, context length (T)=4096, Training Mode=SFT2026.02 | 100 | — | 97.8 | 99.8 | 97.4 | 88.8 | 99.75 | 99.5 | — | |
| RAT+dilation size (D)=8, context length (T)=4096, Training Mode=SFT2026.02 | 100 | — | 96 | 99.8 | 96.4 | 73.8 | 99.4 | 99.15 | — | |
| RAT+dilation size (D)=16, context length (T)=4096, Training Mode=SFT2026.02 | 100 | — | 89.2 | 99.6 | 95.2 | 54.6 | 99.3 | 98.15 | — | |
| RAT+dilation size (D)=32, context length (T)=4096, Training Mode=SFT2026.02 | 100 | — | 78.8 | 99.2 | 78 | 40.4 | 97.35 | 96.35 | — | |
| RAT+dilation size (D)=64, context length (T)=4096, Training Mode=SFT2026.02 | 100 | — | 76 | 98.4 | 46.8 | 30 | 97.35 | 93.9 | — | |
| Mamba-2Model Category=Recurrent models, Context Length=1K2026.05 | 100 | 99.6 | 59.2 | 29 | — | — | — | — | — | |
| Mamba-2Model Category=Recurrent models, Context Length=2K2026.05 | 100 | 99.6 | 38.6 | 21.2 | — | — | — | — | — | |
| Gated DeltaNetModel Category=Recurrent models, Context Length=2K2026.05 | 100 | 100 | 54.2 | 37 | — | — | — | — | — | |
| Gated DeltaNetModel Category=Recurrent models, Context Length=4K2026.05 | 100 | 87.2 | 60.6 | 27.8 | — | — | — | — | — | |
| KDAModel Category=Recurrent models, Context Length=1K2026.05 | 100 | 100 | 77.4 | 54 | — | — | — | — | — | |
| KDAModel Category=Recurrent models, Context Length=2K2026.05 | 100 | 100 | 63.2 | 44.2 | — | — | — | — | — | |
| Mamba-3 (SISO)Model Category=Recurrent models, Context Length=1K2026.05 | 100 | 99.8 | 60.2 | 44.8 | — | — | — | — | — | |
| Mamba-3 (MIMO)Model Category=Recurrent models, Context Length=1K2026.05 | 100 | 99.8 | 89.2 | 49.4 | — | — | — | — | — | |
| Gated DeltaNet-2Model Category=Recurrent models, Context Length=1K2026.05 | 100 | 100 | 92 | 72.6 | — | — | — | — | — | |
| Gated DeltaNet-2Model Category=Recurrent models, Context Length=2K2026.05 | 100 | 100 | 89.8 | 51.4 | — | — | — | — | — | |
| Gated DeltaNet-2Model Category=Recurrent models, Context Length=4K2026.05 | 100 | 93 | 31.8 | 37.8 | — | — | — | — | — | |
| TransformerModel Category=Attention or hybrid models, Context Length=1K2026.05 | 100 | 100 | 95.8 | 75.6 | — | — | — | — | — | |
| TransformerModel Category=Attention or hybrid models, Context Length=2K2026.05 | 100 | 100 | 94.8 | 66.6 | — | — | — | — | — | |
| Mamba-2Model Category=Attention or hybrid models, Context Length=1K2026.05 | 100 | 100 | 97.8 | 82 | — | — | — | — | — | |
| Mamba-2Model Category=Attention or hybrid models, Context Length=2K2026.05 | 100 | 99.6 | 86.8 | 58.6 | — | — | — | — | — | |
| Gated DeltaNetModel Category=Attention or hybrid models, Context Length=1K2026.05 | 100 | 100 | 94.8 | 91 | — | — | — | — | — | |
| Gated DeltaNetModel Category=Attention or hybrid models, Context Length=2K2026.05 | 100 | 99.8 | 91.2 | 78.4 | — | — | — | — | — | |
| KDAModel Category=Attention or hybrid models, Context Length=1K2026.05 | 100 | 100 | 97.2 | 91.4 | — | — | — | — | — | |
| KDAModel Category=Attention or hybrid models, Context Length=2K2026.05 | 100 | 100 | 93.4 | 84 | — | — | — | — | — | |
| Mamba-3 (SISO)Model Category=Attention or hybrid models, Context Length=1K2026.05 | 100 | 100 | 95 | 78.8 | — | — | — | — | — | |
| Mamba-3 (SISO)Model Category=Attention or hybrid models, Context Length=2K2026.05 | 100 | 100 | 90.4 | 65.6 | — | — | — | — | — | |
| Mamba-3 (MIMO)Model Category=Attention or hybrid models, Context Length=1K2026.05 | 100 | 100 | 99.4 | 82.4 | — | — | — | — | — | |
| Mamba-3 (MIMO)Model Category=Attention or hybrid models, Context Length=2K2026.05 | 100 | 100 | 98.4 | 79 | — | — | — | — | — | |
| Gated DeltaNet-2Model Category=Attention or hybrid models, Context Length=1K2026.05 | 100 | 100 | 99.6 | 93 | — | — | — | — | — | |
| Gated DeltaNet-2Model Category=Attention or hybrid models, Context Length=2K2026.05 | 100 | 100 | 99 | 84.6 | — | — | — | — | — | |
| Gated DeltaNetModel Category=Recurrent models, Context Length=1K2026.05 | 99.8 | 100 | 89.8 | 58 | — | — | — | — | — | |
| Mamba-3 (MIMO)Model Category=Recurrent models, Context Length=2K2026.05 | 99.8 | 98.8 | 72.4 | 19.2 | — | — | — | — | — | |
| KDAModel Category=Recurrent models, Context Length=4K2026.05 | 99.2 | 89 | 26.2 | 28 | — | — | — | — | — | |
| Full attentionContext Length=128K, Model Size=3B, Training Context Length=128K2025.12 | 99 | 86 | 64 | — | — | — | — | — | — | |
| Mamba 2Context Length=8K, Model Size=3B, Training Context Length=128K2025.12 | 99 | 99 | 77 | — | — | — | — | — | — | |
| Mamba-3 (SISO)Model Category=Recurrent models, Context Length=2K2026.05 | 99 | 99 | 35.6 | 27.4 | — | — | — | — | — | |
| TTT-KVBContext Length=8K, Model Size=3B, Training Context Length=128K2025.12 | 98 | 100 | 74 | — | — | — | — | — | — | |
| Gated DeltaNet-2Model Category=Recurrent models, Context Length=8K2026.05 | 97.8 | 39.2 | — | — | — | — | — | — | — | |
| Gated DeltaNetModel Category=Recurrent models, Context Length=8K2026.05 | 97.6 | 32 | — | — | — | — | — | — | — | |
| Mamba-2Model Category=Recurrent models, Context Length=4K2026.05 | 97 | 62.6 | 14.4 | 21.4 | — | — | — | — | — | |
| Hybrid SWA and fullContext Length=16K, Model Size=3B, Training Context Length=128K2025.12 | 93 | 100 | 56 | — | — | — | — | — | — | |
| Mamba-3 (MIMO)Model Category=Recurrent models, Context Length=4K2026.05 | 93 | 64.2 | 29.2 | 18 | — | — | — | — | — | |
| Hybrid SWA and fullContext Length=32K, Model Size=3B, Training Context Length=128K2025.12 | 88 | 99 | 32 | — | — | — | — | — | — | |
| KDAModel Category=Recurrent models, Context Length=8K2026.05 | 70.6 | 30.6 | — | — | — | — | — | — | — | |
| Hybrid SWA and fullContext Length=64K, Model Size=3B, Training Context Length=128K2025.12 | 69 | 89 | 17 | — | — | — | — | — | — | |
| Mamba-3 (SISO)Model Category=Recurrent models, Context Length=4K2026.05 | 63.4 | 59.4 | 12.2 | 20.2 | — | — | — | — | — | |
| Mamba-2Model Category=Recurrent models, Context Length=8K2026.05 | 55.8 | 21 | — | — | — | — | — | — | — | |
| Gated DeltaNet-2Model Category=Attention or hybrid models, Context Length=4K2026.05 | 55.2 | 57.9 | 55.6 | 48 | — | — | — | — | — | |
| Mamba-2Model Category=Attention or hybrid models, Context Length=4K2026.05 | 51.8 | 52.4 | 48 | 39 | — | — | — | — | — | |
| KDAModel Category=Attention or hybrid models, Context Length=4K2026.05 | 51.8 | 56 | 51.6 | 40.4 | — | — | — | — | — | |
| TransformerModel Category=Attention or hybrid models, Context Length=4K2026.05 | 51.2 | 44.2 | 37 | 38.2 | — | — | — | — | — | |
| SWAContext Length=16K, Model Size=3B, Training Context Length=128K2025.12 | 50 | 43 | 41 | — | — | — | — | — | — | |
| Gated DeltaNetContext Length=16K, Model Size=3B, Training Context Length=128K2025.12 | 50 | 43 | 45 | — | — | — | — | — | — | |
| Mamba-3 (SISO)Model Category=Attention or hybrid models, Context Length=4K2026.05 | 49.6 | 58.2 | 44 | 33.6 | — | — | — | — | — | |
| Mamba 2Context Length=16K, Model Size=3B, Training Context Length=128K2025.12 | 49 | 43 | 36 | — | — | — | — | — | — | |
| Mamba-3 (MIMO)Model Category=Attention or hybrid models, Context Length=4K2026.05 | 49 | 53 | 54.2 | 46.6 | — | — | — | — | — | |
| Gated DeltaNetModel Category=Attention or hybrid models, Context Length=4K2026.05 | 47.2 | 57.3 | 47.2 | 44.8 | — | — | — | — | — | |
| TTT-E2EContext Length=16K, Model Size=3B, Training Context Length=128K2025.12 | 46 | 43 | 44 | — | — | — | — | — | — | |
| TTT-KVBContext Length=16K, Model Size=3B, Training Context Length=128K2025.12 | 43 | 43 | 34 | — | — | — | — | — | — | |
| Mamba-3 (MIMO)Model Category=Recurrent models, Context Length=8K2026.05 | 35.6 | 27.2 | — | — | — | — | — | — | — | |
| Mamba-3 (SISO)Model Category=Recurrent models, Context Length=8K2026.05 | 27.8 | 25.2 | — | — | — | — | — | — | — | |
| Gated DeltaNet-2Model Category=Attention or hybrid models, Context Length=8K2026.05 | 27.4 | 29.2 | — | — | — | — | — | — | — | |
| KDAModel Category=Attention or hybrid models, Context Length=8K2026.05 | 26.2 | 23 | — | — | — | — | — | — | — | |
| SWAContext Length=32K, Model Size=3B, Training Context Length=128K2025.12 | 26 | 28 | 24 | — | — | — | — | — | — | |
| Mamba 2Context Length=32K, Model Size=3B, Training Context Length=128K2025.12 | 26 | 28 | 24 | — | — | — | — | — | — | |
| Gated DeltaNetContext Length=32K, Model Size=3B, Training Context Length=128K2025.12 | 26 | 27 | 23 | — | — | — | — | — | — | |
| Mamba-3 (SISO)Model Category=Attention or hybrid models, Context Length=8K2026.05 | 26 | 27.8 | — | — | — | — | — | — | — | |
| Mamba-2Model Category=Attention or hybrid models, Context Length=8K2026.05 | 25.4 | 25.8 | — | — | — | — | — | — | — | |
| TTT-E2EContext Length=32K, Model Size=3B, Training Context Length=128K2025.12 | 24 | 28 | 24 | — | — | — | — | — | — | |
| Mamba-3 (MIMO)Model Category=Attention or hybrid models, Context Length=8K2026.05 | 22.8 | 27.8 | — | — | — | — | — | — | — | |
| Gated DeltaNetModel Category=Attention or hybrid models, Context Length=8K2026.05 | 22.4 | 25.6 | — | — | — | — | — | — | — | |
| TTT-KVBContext Length=32K, Model Size=3B, Training Context Length=128K2025.12 | 22 | 27 | 23 | — | — | — | — | — | — | |
| Hybrid SWA and fullContext Length=128K, Model Size=3B, Training Context Length=128K2025.12 | 21 | 29 | 6 | — | — | — | — | — | — | |
| SWAContext Length=64K, Model Size=3B, Training Context Length=128K2025.12 | 13 | 16 | 9 | — | — | — | — | — | — | |
| Mamba 2Context Length=64K, Model Size=3B, Training Context Length=128K2025.12 | 13 | 16 | 8 | — | — | — | — | — | — | |
| Gated DeltaNetContext Length=64K, Model Size=3B, Training Context Length=128K2025.12 | 13 | 16 | 7 | — | — | — | — | — | — | |
| TTT-E2EContext Length=64K, Model Size=3B, Training Context Length=128K2025.12 | 13 | 16 | 10 | — | — | — | — | — | — | |
| TTT-KVBContext Length=64K, Model Size=3B, Training Context Length=128K2025.12 | 10 | 16 | 6 | — | — | — | — | — | — | |
| SWAContext Length=128K, Model Size=3B, Training Context Length=128K2025.12 | 7 | 5 | 5 | — | — | — | — | — | — | |
| Mamba 2Context Length=128K, Model Size=3B, Training Context Length=128K2025.12 | 7 | 5 | 4 | — | — | — | — | — | — | |
| Gated DeltaNetContext Length=128K, Model Size=3B, Training Context Length=128K2025.12 | 7 | 5 | 3 | — | — | — | — | — | — | |
| TTT-E2EContext Length=128K, Model Size=3B, Training Context Length=128K2025.12 | 6 | 5 | 3 | — | — | — | — | — | — | |
| TTT-KVBContext Length=128K, Model Size=3B, Training Context Length=128K2025.12 | 1 | 5 | 4 | — | — | — | — | — | — | |
| TransformerModel Category=Attention or hybrid models, Context Length=8K2026.05 | 0 | 0 | — | — | — | — | — | — | — |