Long-Context Retrieval on RULER (4K to 256K Sweep)
100Retrieval Accuracy (4K Context)Qwen3-1.7B
Evaluation results
| Method | Links | |||||||
|---|---|---|---|---|---|---|---|---|
| Qwen3-1.7BType=FA2026.06 | 100 | 99.87 | 99.93 | 99.93 | 98.13 | 0 | 0 | |
| Qwen3-1.7B-yarnType=FA2026.06 | 100 | 99.93 | 98.93 | 97.73 | 98.8 | 83.47 | 40.2 | |
| Jet-Nemotron-2BType=Hybrid2026.06 | 100 | 99.67 | 99.47 | 92.8 | 27.27 | 12.6 | — | |
| HydraHeadType=Hybrid2026.06 | 100 | 99.93 | 100 | 99.8 | 99.8 | 99.47 | 94.53 | |
| MiniCPM5-1B-BaseType=FA2026.06 | 99.93 | 99.47 | 98.6 | 97.6 | 90.93 | 93.73 | 59.53 | |
| Gemma-3n-E2BType=Hybrid2026.06 | 99.93 | 99.87 | 99.67 | 99.67 | 22.23 | 0 | 0 | |
| INF-V2Full name=InfLLM-v2, Attention=Sparse attention baseline, Block selection=First-order Taylor expansion, Parameters=14B, Block size (B)=1282026.01 | 95.9 | 94.1 | 92.1 | 89.3 | 86.7 | 61.6 | 42.6 | |
| SPADescription=Ablation variant of SPLA, Block selection=Second-order Taylor-based, Residual Linear Attention=Excluded, Parameters=14B, Block size (B)=1282026.01 | 95.9 | 94.2 | 93.3 | 90.4 | 87.1 | 62.4 | 44.5 | |
| SPLAFull name=Block Sparse Plus Linear Attention, Block selection=Second-order Taylor-based, Residual Linear Attention=Included, Parameters=14B, Block size (B)=1282026.01 | 95.9 | 94.7 | 94.2 | 91.7 | 88.3 | 85.2 | 72.3 | |
| DENSEAttention=Full-attention baseline, Parameters=14B2026.01 | 95.8 | 94.9 | 93.6 | 91.4 | 87.1 | 83.2 | 69.3 | |
| Hymba-1.5BType=Hybrid2026.06 | 95.6 | 88.2 | 25.6 | 0.07 | 0 | 0 | 0 | |
| HypeNet-2BType=Hybrid2026.06 | 93.6 | 94.53 | 86.33 | 84.33 | 83.93 | 78.8 | 68.93 | |
| NSAAttention=Sparse attention baseline, Parameters=14B, Block size (B)=1282026.01 | 92.3 | 91.4 | 90.7 | 84.6 | 83.2 | 51.3 | 32.5 |