Long-Context Retrieval on RULER (4K to 128K Sweep)
98.14Retrieval Accuracy (8K)Full-KV
Evaluation results
| Method | Links | |||||||
|---|---|---|---|---|---|---|---|---|
| Full-KVBackbone=Llama-3.1-8B-Instruct2026.05 | 98.14 | 98.93 | 97.87 | 94.34 | 89.32 | 78.89 | 92.92 | |
| MrRoPE-ProBase Model=LLaMA3-8B-Instruct, Scale Setting=8K → 128K2026.01 | 96.2 | — | 94.2 | 94.3 | 91.3 | 86.6 | — | |
| YaRNBase Model=LLaMA3-8B-Instruct, Scale Setting=8K → 128K2026.01 | 95.5 | — | 92.1 | 92.7 | 89.5 | 79.9 | — | |
| Nemo. 3 Nano# Parameters=30B A3B, # Train Tokens=25T, Train Compute (10^23 FLOPs)=5.42026.04 | 95.2 | 96.2 | 93.8 | 90.3 | 85.9 | — | — | |
| Dense AttentionModel=Llama-3.1-8B-Instruct, Phase=Prefill2025.12 | 95.07 | 96.16 | 94.8 | 92.33 | 87.69 | — | 93.21 | |
| BLASSTModel=Llama-3.1-8B-Instruct, Phase=Prefill, Sparsity=~50%2025.12 | 94.7 | 96.17 | 94.61 | 91.81 | 87.06 | — | 92.87 | |
| PrismBackbone=Llama-3.1-8B2026.02 | 94.47 | 95.28 | 92.48 | 87.67 | 82.59 | 72.75 | 87.54 | |
| XAttentionModel=Llama-3.1-8B-Instruct, Phase=Prefill2025.12 | 94.47 | 96.37 | 94.48 | 91.91 | 85.01 | — | 92.44 | |
| MInferenceBackbone=Llama-3.1-8B2026.02 | 94.46 | 95.43 | 93.42 | 87.22 | 83.07 | 71.04 | 87.44 | |
| FullBackbone=Llama-3.1-8B2026.02 | 94.38 | 95.42 | 93.38 | 87.98 | 84.72 | 77.77 | 88.94 | |
| XAttentionBackbone=Llama-3.1-8B2026.02 | 94.3 | 95.17 | 93.28 | 89.06 | 82.31 | 70.52 | 87.44 | |
| Qwen 3# Parameters=8B, # Train Tokens=36T, Train Compute (10^23 FLOPs)=17.32026.04 | 94.1 | 95.5 | 93.5 | 90.1 | — | — | — | |
| MInferenceModel=Llama-3.1-8B-Instruct, Phase=Prefill2025.12 | 94.06 | 96.54 | 91.37 | 85.79 | 83.03 | — | 84.15 | |
| FlexPrefillModel=Llama-3.1-8B-Instruct, Phase=Prefill2025.12 | 93.67 | 95.99 | 92.73 | 88.14 | 81.14 | — | 87.72 | |
| FlexPrefillBackbone=Llama-3.1-8B2026.02 | 92.44 | 93.8 | 93.28 | 87.92 | 84.74 | 72.41 | 87.43 | |
| MInferenceBackbone=Qwen-3-8B(YaRN)2026.02 | 92.37 | 95.08 | 89.67 | 86.01 | 76.53 | 70.36 | 85 | |
| FullBackbone=Qwen-3-8B(YaRN)2026.02 | 92.35 | 95.01 | 90.04 | 87.24 | 79.93 | 75.09 | 86.61 | |
| Falcon H1# Parameters=7B, # Train Tokens=12T, Train Compute (10^23 FLOPs)=5.52026.04 | 92 | 92 | 89.7 | 83 | 77.2 | — | — | |
| Olmo Hybrid# Parameters=7B, # Train Tokens=6T, Train Compute (10^23 FLOPs)=2.6, Decoding Settings=--disable-cascade-attn, --enforce-eager2026.04 | 91.4 | 92.7 | 90 | 86.4 | 85 | — | — | |
| Olmo 3# Parameters=7B, # Train Tokens=6T, Train Compute (10^23 FLOPs)=2.6, Decoding Settings=--disable-cascade-attn, --enforce-eager2026.04 | 91.2 | 94.9 | 84.1 | 78.3 | 67.8 | — | — | |
| XAttentionBackbone=Qwen-3-8B(YaRN)2026.02 | 91.03 | 94.55 | 87.91 | 84.37 | 77.73 | 72.01 | 84.6 | |
| PrismBackbone=Qwen-3-8B(YaRN)2026.02 | 90.95 | 94.84 | 87.69 | 86.88 | 78.58 | 72.65 | 85.27 | |
| Full-contextPrefill=100%, KV=100%2026.04 | 90.1 | — | 95 | 83.4 | 85.5 | 76.3 | 86 | |
| Kimi Linear# Parameters=48B A3B, # Train Tokens=6T, Train Compute (10^23 FLOPs)=1.02026.04 | 89.4 | 88.5 | 94 | 90.2 | 90.5 | — | — | |
| FlexPrefillBackbone=Qwen-3-8B(YaRN)2026.02 | 87.61 | 90.89 | 87.82 | 85.58 | 78.27 | 73.42 | 83.93 | |
| Meta-SoftBackbone=Llama-3.1-8B-Instruct2026.05 | 86.15 | 93.81 | 79.67 | 74.08 | 65.58 | 55.03 | 75.72 | |
| AnDProBackbone=Llama-3.1-8B-Instruct2026.05 | 85.53 | 92.23 | 78.77 | 73.36 | 64.94 | 54.28 | 74.85 | |
| Judge QBackbone=Llama-3.1-8B-Instruct2026.05 | 84.63 | 91.12 | 77.65 | 72.19 | 64.27 | 53.89 | 73.96 | |
| ZeroMergeBackbone=Llama-3.1-8B-Instruct2026.05 | 83.71 | 90.29 | 76.52 | 71.44 | 62.95 | 51.94 | 72.81 | |
| Nemotron-H# Parameters=8B, # Train Tokens=15T, Train Compute (10^23 FLOPs)=7.22026.04 | 82 | 84.1 | — | — | — | — | — | |
| StructKVPrefill=60%, KV=10%2026.04 | 81.3 | — | 82.5 | 81.8 | 81.5 | 73.6 | 80.1 | |
| FastKVPrefill=60%, KV=10%2026.04 | 77.8 | — | 77.3 | 77.2 | 77.4 | 68.2 | 75.6 | |
| LAQBackbone=Llama-3.1-8B-Instruct2026.05 | 77.18 | 85.23 | 73.24 | 68.09 | 58.91 | 49.04 | 68.62 | |
| CaMBackbone=Llama-3.1-8B-Instruct2026.05 | 76.98 | 85.01 | 72.83 | 67.81 | 57.82 | 48.76 | 68.2 | |
| SnapKVPrefill=100%, KV=10%2026.04 | 75.6 | — | 76.8 | 72.9 | 75 | 67.7 | 73.6 | |
| SnapKVBackbone=Llama-3.1-8B-Instruct2026.05 | 75.49 | 83.34 | 71.07 | 66.72 | 57.26 | 47.65 | 66.92 | |
| GemFilterPrefill=51%, KV=10%2026.04 | 69.7 | — | 68.2 | 70.4 | 69.8 | 69.8 | 69.6 | |
| PyramidInferPrefill=60%, KV=60%2026.04 | 66.5 | — | — | — | — | — | — | |
| H2OBackbone=Llama-3.1-8B-Instruct2026.05 | 53.45 | 59.46 | 47.79 | 42.87 | 32.54 | 23.11 | 43.2 | |
| Falcon Mamba# Parameters=7B, # Train Tokens=6T, Train Compute (10^23 FLOPs)=2.52026.04 | 49 | 68.2 | 23.8 | 4.7 | 0 | — | — | |
| H2OPrefill=100%, KV=10%2026.04 | 27.1 | — | — | — | — | — | — | |
| StreamingLLMBackbone=Llama-3.1-8B-Instruct2026.05 | 20.01 | 39.87 | 12.13 | 10.59 | 9.97 | 8.23 | 16.8 | |
| xLSTM# Parameters=7B, # Train Tokens=2T, Train Compute (10^23 FLOPs)=0.92026.04 | 20 | 28.6 | 12.4 | — | — | — | — | |
| StreamingLLMPrefill=100%, KV=10%2026.04 | 15 | — | 21.5 | 24.4 | 16.9 | 15 | 18.6 | |
| Recurr. Gemma# Parameters=9B, # Train Tokens=2T, Train Compute (10^23 FLOPs)=1.02026.04 | — | 44.9 | — | — | — | — | — |