Long-context understanding on RULER (4K to 128K Contexts)
157Performance @ 4K ContextPyramidInfer
Evaluation Results
| Method | Links | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| PyramidInferBackbone=Llama-3.1-8B-UL, KV Budget=20482026.01 | 157 | 163 | — | — | — | — | — | — | — | |
| PyramidInferBackbone=Qwen2.5-7B, KV Budget=20482026.01 | 139 | 132 | — | — | — | — | — | — | — | |
| ASLBackbone=Qwen2.5-7B, KV Budget=20482026.01 | 101 | 89 | 72 | 49 | 30 | 15 | — | — | — | |
| FastKVBackbone=Qwen2.5-7B, KV Budget=20482026.01 | 99 | 88 | 71 | 48 | 30 | 15 | — | — | — | |
| SnapKVBackbone=Llama-3.1-8B-UL, KV Budget=20482026.01 | 98 | 94 | 83 | 68 | 48 | 30 | — | — | — | |
| SnapKVBackbone=Qwen2.5-7B, KV Budget=20482026.01 | 98 | 87 | 70 | 48 | 29 | 15 | — | — | — | |
| UniSparseModel=Llama-3.1-8B-Instruct, Top-P threshold=0.9, Sparsity=58.90%2025.12 | 97.38 | 96.59 | 96.43 | 94.25 | 89.29 | 80.73 | 92.44 | — | — | |
| FlexPrefillModel=Llama-3.1-8B-Instruct, Top-P threshold=0.99, Sparsity=36.51%2025.12 | 97.33 | 96.79 | 96.35 | 93.99 | 90.53 | 80.52 | 92.59 | — | — | |
| FlashAttentionModel=Llama-3.1-8B-Instruct, Sparsity=0.0%2025.12 | 97.3 | 96.95 | 96.42 | 93.96 | 89.96 | 82.65 | 92.87 | — | — | |
| MInferenceModel=Llama-3.1-8B-Instruct, Sparsity=50.39%2025.12 | 97.29 | 96.69 | 96.71 | 93.4 | 88.67 | 81.38 | 92.35 | — | — | |
| XAttentionModel=Llama-3.1-8B-Instruct, Top-P threshold=0.95, Sparsity=44.33%2025.12 | 97.27 | 96.49 | 96.15 | 93.82 | 90.16 | 81.34 | 92.54 | — | — | |
| XAttentionModel=Llama-3.1-8B-Instruct, Top-P threshold=0.9, Sparsity=56.44%2025.12 | 97.21 | 96.57 | 95.85 | 93.73 | 89.02 | 79.17 | 91.93 | — | — | |
| UniSparseModel=Llama-3.1-8B-Instruct, Top-P threshold=0.95, Sparsity=46.33%2025.12 | 97.16 | 96.52 | 96.31 | 94.03 | 89.4 | 81.94 | 92.56 | — | — | |
| Qwen2.5-14B-Instruct-1MContext Window=1M2025.12 | 96.6 | 95.5 | 94.3 | 94 | 92.7 | 77.3 | — | — | — | |
| Qwen2.5-72B2025.12 | 96.3 | 94.7 | 93.2 | 90.1 | 74.7 | 53.9 | — | — | — | |
| Llama3.1-8b-instructType=Instruct2025.12 | 96.2 | 94.8 | 94.4 | 92.5 | 86.4 | 80.7 | — | — | — | |
| Flash Attention w/ Token SparseBackbone=LLaMA-3.1-8B-Instruct2026.02 | 96.06 | 92.9 | 91.82 | 84.81 | 82.83 | 73.68 | 87.02 | 1.36 | — | |
| Flash AttentionBackbone=LLaMA-3.1-8B-Instruct2026.02 | 95.82 | 92.77 | 91.02 | 84.87 | 83.43 | 74.15 | 87.01 | 1 | — | |
| FlexPrefillModel=Llama-3.1-8B-Instruct, Top-P threshold=0.95, Sparsity=64.16%2025.12 | 95.58 | 96.29 | 95.79 | 94.38 | 90.89 | 81.31 | 92.37 | — | — | |
| FlexPrefillBackbone=LLaMA-3.1-8B-Instruct2026.02 | 95.48 | 92.71 | 91.4 | 87.2 | 83.05 | 73.75 | 87.27 | 2.44 | — | |
| FlexPrefill w/ Token SparseBackbone=LLaMA-3.1-8B-Instruct2026.02 | 95.33 | 92.47 | 91.49 | 87.68 | 83.07 | 73.58 | 87.27 | 2.76 | — | |
| Flash AttentionBackbone=Mistral-Nemo-12B-Instruct2026.02 | 95.19 | 92.29 | 85.6 | 64.86 | 47.98 | 19.68 | 67.6 | 1 | — | |
| Flash Attention w/ Token SparseBackbone=Mistral-Nemo-12B-Instruct2026.02 | 95.07 | 92.1 | 85.97 | 63.67 | 47.97 | 19.44 | 67.37 | 1.22 | — | |
| FlexPrefill w/ Token SparseBackbone=Mistral-Nemo-12B-Instruct2026.02 | 94.84 | 93.31 | 86.14 | 64.89 | 48.7 | 19.58 | 67.91 | 1.33 | — | |
| FlexPrefillBackbone=Mistral-Nemo-12B-Instruct2026.02 | 94.79 | 93.13 | 86.62 | 64.58 | 49.3 | 20.54 | 68.16 | 1.22 | — | |
| FlashAttentionModel=Qwen2.5-7B-Instruct, Sparsity=0.0%2025.12 | 94.77 | 89.42 | 87.84 | 85.46 | 81.18 | 70.92 | 84.93 | — | — | |
| MInferenceModel=Qwen2.5-7B-Instruct, Sparsity=39.32%2025.12 | 94.68 | 89.65 | 88.56 | 85.04 | 78.47 | 65.08 | 83.58 | — | — | |
| Qwen2.5-72BExtension=YaRN2025.12 | 94.6 | 90.7 | 89.1 | 87.1 | 81.7 | 75 | — | — | — | |
| UniSparseModel=Qwen2.5-7B-Instruct, Top-P threshold=0.95, Sparsity=47.25%2025.12 | 94.31 | 88.96 | 87.19 | 85.23 | 80.42 | 69.47 | 84.26 | — | — | |
| mid-2Checkpoint Stage=mid-22025.12 | 94.3 | 92.2 | 89.2 | 73.4 | 51.6 | 12 | — | — | — | |
| Full KVBackbone=Llama-3.1-8B-UL, KV Budget=Full2026.01 | 94.2 | 92 | 89.5 | 81.1 | 73.1 | 68.6 | — | — | — | |
| ASLBackbone=Llama-3.1-8B-UL, KV Budget=Full (before selection) & 2048 (after selection)2026.01 | 94.2 | 90.5 | 87.2 | 80 | 70.5 | 69.2 | — | — | — | |
| Full KVBackbone=Qwen2.5-7B, KV Budget=Full2026.01 | 94 | 93 | 92.7 | 89.6 | 86.5 | 82.3 | — | — | — | |
| ASLBackbone=Qwen2.5-7B, KV Budget=Full (before selection) & 2048 (after selection)2026.01 | 94 | 91.2 | 88 | 83.6 | 80.9 | 74.2 | — | — | — | |
| mid-4Checkpoint Stage=mid-42025.12 | 93.9 | 93.2 | 92.9 | 89.6 | 85.3 | 74.6 | — | — | — | |
| XAttentionModel=Qwen2.5-7B-Instruct, Top-P threshold=0.95, Sparsity=46.44%2025.12 | 93.87 | 88.36 | 86.84 | 84.15 | 79.04 | 69.51 | 83.63 | — | — | |
| SnapKVBackbone=Llama-3.1-8B-UL, KV Budget=20482026.01 | 93.8 | 88.3 | 81 | 74.8 | 63.6 | 55.6 | — | — | — | |
| ASLBackbone=Llama-3.1-8B-UL, KV Budget=20482026.01 | 93.7 | 87 | 79.7 | 73 | 63.2 | 56.1 | — | — | — | |
| FastKVBackbone=Llama-3.1-8B-UL, KV Budget=Full (before selection) & 2048 (after selection)2026.01 | 93.7 | 87.3 | 82.9 | 73 | 63.4 | 60.6 | — | — | — | |
| MinferenceBackbone=LLaMA-3.1-8B-Instruct2026.02 | 93.46 | 92.29 | 91.01 | 85.34 | 83.19 | 73.63 | 86.49 | 1.12 | — | |
| FastKVBackbone=Llama-3.1-8B-UL, KV Budget=20482026.01 | 93.4 | 85.8 | 79.3 | 69.5 | 60.1 | 55.1 | — | — | — | |
| UniSparseModel=Qwen2.5-7B-Instruct, Top-P threshold=0.9, Sparsity=60.12%2025.12 | 93.23 | 87.69 | 85.99 | 84.26 | 79.25 | 68 | 83.07 | — | — | |
| mid-3Checkpoint Stage=mid-32025.12 | 93.1 | 92.4 | 91.1 | 88.1 | 82.1 | 67 | — | — | — | |
| Minference w/ Token SparseBackbone=LLaMA-3.1-8B-Instruct2026.02 | 93.05 | 92 | 91.03 | 85.1 | 82.92 | 72.18 | 86.05 | 1.38 | — | |
| Minference w/ Token SparseBackbone=Mistral-Nemo-12B-Instruct2026.02 | 93.01 | 91.36 | 84.31 | 64.7 | 46.67 | 18.68 | 66.46 | 1.28 | — | |
| XAttentionModel=Qwen2.5-7B-Instruct, Top-P threshold=0.9, Sparsity=58.17%2025.12 | 92.84 | 87.91 | 85.02 | 82.8 | 77.14 | 66.58 | 82.05 | — | — | |
| MinferenceBackbone=Mistral-Nemo-12B-Instruct2026.02 | 92.52 | 91.02 | 84.29 | 65.18 | 46.52 | 19 | 66.42 | 1.13 | — | |
| GemFilterBackbone=Llama-3.1-8B-UL, KV Budget=20482026.01 | 92.5 | 81.1 | 79 | 76 | 71 | 66.7 | — | — | — | |
| FlexPrefillModel=Qwen2.5-7B-Instruct, Top-P threshold=0.99, Sparsity=37.75%2025.12 | 92.42 | 88.77 | 85.91 | 84.13 | 77.89 | 67.69 | 82.8 | — | — | |
| FastKVBackbone=Qwen2.5-7B, KV Budget=Full (before selection) & 2048 (after selection)2026.01 | 91.3 | 80.6 | 75.8 | 70.6 | 64.2 | 59.1 | — | — | — | |
| FastKVBackbone=Llama-3.1-8B-UL, KV Budget=20482026.01 | 91 | 85 | 77 | 62 | 43 | 27 | — | — | — | |
| FastKVBackbone=Qwen2.5-7B, KV Budget=20482026.01 | 91 | 79.7 | 75.3 | 69.9 | 63.2 | 59.1 | — | — | — | |
| SnapKVBackbone=Qwen2.5-7B, KV Budget=20482026.01 | 89.9 | 77.9 | 75.3 | 71.8 | 66.8 | 65 | — | — | — | |
| ASLBackbone=Qwen2.5-7B, KV Budget=20482026.01 | 89.9 | 77.5 | 75.5 | 75.1 | 71.7 | 66.4 | — | — | — | |
| ASLBackbone=Llama-3.1-8B-UL, KV Budget=20482026.01 | 89 | 86 | 77 | 62 | 44 | 28 | — | — | — | |
| FlexPrefillModel=Qwen2.5-7B-Instruct, Top-P threshold=0.95, Sparsity=64.14%2025.12 | 83.78 | 81.45 | 80.2 | 76.5 | 71.21 | 58.06 | 75.2 | — | — | |
| ASL_2passBackbone=Llama-3.1-8B-UL, KV Budget=20482026.01 | 82.2 | 74.6 | 74.8 | 78.5 | 69.7 | 54.5 | — | — | — | |
| GemFilterBackbone=Llama-3.1-8B-UL, KV Budget=20482026.01 | 81 | 80 | 71 | 58 | 40 | 25 | — | — | — | |
| ASL_2passBackbone=Llama-3.1-8B-UL, KV Budget=20482026.01 | 80 | 77 | 69 | 57 | 41 | 25 | — | — | — | |
| GemFilterBackbone=Qwen2.5-7B, KV Budget=20482026.01 | 79.7 | 67.7 | 66.9 | 63.8 | 60.4 | 56.7 | — | — | — | |
| ASL_2passBackbone=Qwen2.5-7B, KV Budget=20482026.01 | 78.3 | 70.7 | 67.9 | 71.5 | 69.1 | 62.5 | — | — | — | |
| PyramidInferBackbone=Llama-3.1-8B-UL, KV Budget=20482026.01 | 76.4 | 74.7 | — | — | — | — | — | — | — | |
| GemFilterBackbone=Qwen2.5-7B, KV Budget=20482026.01 | 76 | 68 | 55 | 37 | 23 | 11 | — | — | — | |
| ASL_2passBackbone=Qwen2.5-7B, KV Budget=20482026.01 | 76 | 72 | 62 | 55 | 23 | 13 | — | — | — | |
| PyramidInferBackbone=Qwen2.5-7B, KV Budget=20482026.01 | 37.9 | 30.2 | — | — | — | — | — | — | — | |
| CSDModel Pair=Llama-3-70B/1B2026.04 | — | — | — | — | — | — | 95.9 | 1.37 | 83.2 | |
| Standard SDModel Pair=Llama-3-70B/1B2026.04 | — | — | — | — | — | — | 88 | 1.02 | 73.6 | |
| VanillaModel Pair=Llama-3-70B/1B2026.04 | — | — | — | — | — | — | 88.1 | 1 | — |