Long-Context Language Modeling Performance and Speedup on RULER
97.36Score (4K)Baseline
Evaluation results
| Method | Links | ||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| BaselineModel=Llama-3.1-8B-Instruct, Attention Type=Full Attention, Batch Size=12026.05 | 97.36 | 95.98 | 94.62 | 91.02 | 86.29 | 76.89 | 90.36 | 1 | 1 | 1 | 1 | 1 | 1 | — | |
| FlashPrefillBackbone Model=Llama-3.1-8B-Instruct2026.03 | 97.27 | 96.2 | 94.97 | 92.21 | 84.93 | 75.31 | 90.15 | 1.71 | 2.81 | 4.63 | 7.48 | 13.62 | 22.67 | — | |
| BaselineModel=Qwen3-Next-80B-A3B, Attention Type=Linear/Full Attention Hybrid, Batch Size=12026.05 | 96.83 | 95.67 | 95.07 | 94.38 | 94.51 | 92.09 | 94.76 | 1 | 1 | 1 | 1 | 1 | 1 | — | |
| ProxyAttnModel=Llama-3.1-8B-Instruct, Attention Type=Full Attention, Batch Size=12026.05 | 96.78 | 95.46 | 95.49 | 89.28 | 85.31 | 78.49 | 90.14 | 0.82 | 0.89 | 1.03 | 1.11 | 1.36 | 1.79 | — | |
| MInferenceModel=Llama-3.1-8B-Instruct, Attention Type=Full Attention, Batch Size=12026.05 | 96.71 | 95.78 | 95.51 | 90.76 | 87.12 | 78.21 | 90.68 | 0.82 | 0.86 | 0.98 | 1.03 | 1.15 | 1.34 | — | |
| UniPrefillModel=Qwen3-Next-80B-A3B, Attention Type=Linear/Full Attention Hybrid, Batch Size=12026.05 | 96.67 | 94.49 | 94.29 | 93.63 | 93.13 | 91.41 | 93.94 | 1.08 | 1.21 | 1.24 | 1.39 | 1.42 | 1.68 | — | |
| MInferenceModel=Qwen3-Next-80B-A3B, Attention Type=Linear/Full Attention Hybrid, Batch Size=12026.05 | 96.62 | 94.38 | 94.49 | 94.27 | 94.28 | 91.81 | 94.31 | 0.96 | 0.98 | 1 | 1 | 1.02 | 1.05 | — | |
| UniPrefillModel=Llama-3.1-8B-Instruct, Attention Type=Full Attention, Batch Size=12026.05 | 96.53 | 95.83 | 95.41 | 89.77 | 85.28 | 79.87 | 90.45 | 1.21 | 1.34 | 1.37 | 1.62 | 2.01 | 2.26 | — | |
| FlexPrefillModel=Qwen3-Next-80B-A3B, Attention Type=Linear/Full Attention Hybrid, Batch Size=12026.05 | 96.36 | 95.03 | 94.17 | 93.91 | 92.89 | 91.44 | 93.97 | 0.96 | 0.98 | 1 | 1.01 | 1.04 | 1.08 | — | |
| MInferenceBackbone Model=Llama-3.1-8B-Instruct2026.03 | 96.34 | 93.79 | 93.48 | 89.76 | 84.32 | 71.28 | 88.16 | 0.11 | 0.16 | 0.45 | 0.83 | 1.34 | 2.45 | — | |
| FlexPrefillModel=Llama-3.1-8B-Instruct, Attention Type=Full Attention, Batch Size=12026.05 | 96.34 | 95.12 | 94.83 | 88.96 | 84.31 | 78.13 | 89.62 | 0.83 | 0.89 | 1.02 | 1.08 | 1.24 | 1.46 | — | |
| ProxyAttnModel=Qwen3-Next-80B-A3B, Attention Type=Linear/Full Attention Hybrid, Batch Size=12026.05 | 96.31 | 94.13 | 94.23 | 93.47 | 93.51 | 91.62 | 93.88 | 0.96 | 0.98 | 1 | 1.02 | 1.05 | 1.11 | — | |
| Full AttentionBackbone Model=Llama-3.1-8B-Instruct2026.03 | 96.07 | 93.94 | 93.51 | 90.68 | 86.29 | 74.25 | 89.12 | 1 | 1 | 1 | 1 | 1 | 1 | — | |
| XAttentionModel=Qwen3-Next-80B-A3B, Attention Type=Linear/Full Attention Hybrid, Batch Size=12026.05 | 96.03 | 94.81 | 94.03 | 93.01 | 93.06 | 90.23 | 93.53 | 0.97 | 0.99 | 1 | 1 | 1.02 | 1.05 | — | |
| XAttentionModel=Llama-3.1-8B-Instruct, Attention Type=Full Attention, Batch Size=12026.05 | 95.98 | 95.23 | 94.68 | 88.06 | 83.92 | 78.16 | 89.34 | 0.92 | 0.96 | 1.03 | 1.08 | 1.21 | 1.38 | — | |
| Llama-3.1-8B-InstructQuantization=4-bit, Token Density=12.5%2025.10 | 95.8 | 93.8 | 93.2 | 81.5 | — | — | — | — | — | — | — | — | — | 75.2 | |
| Full AttentionBackbone Model=Qwen3-30B-A3B-Instruct-25072026.03 | 95.78 | 95.79 | 95.82 | 94.31 | 90.25 | 87.71 | 93.28 | 1 | 1 | 1 | 1 | 1 | 1 | — | |
| FlashPrefillBackbone Model=Qwen3-30B-A3B-Instruct-25072026.03 | 95.76 | 95.89 | 95.03 | 94.06 | 90.11 | 85.2 | 92.68 | 1.71 | 2.79 | 4.48 | 6.94 | 11.45 | 18.67 | — | |
| FlexPrefillBackbone Model=Qwen3-30B-A3B-Instruct-25072026.03 | 95.67 | 95.42 | 95.23 | 93.62 | 89.62 | 86.12 | 92.61 | 0.12 | 0.34 | 0.98 | 2.31 | 3.92 | 5.43 | — | |
| FlexPrefillBackbone Model=Llama-3.1-8B-Instruct2026.03 | 95.63 | 93.14 | 91.92 | 90.36 | 85.49 | 73.23 | 88.3 | 0.11 | 0.33 | 0.98 | 2.21 | 4.16 | 5.18 | — | |
| MInferenceBackbone Model=Qwen2.5-7B-Instruct2026.03 | 95.61 | 93.21 | 91.62 | 90.16 | 68.22 | 24.38 | 77.2 | 0.11 | 0.16 | 0.45 | 0.86 | 1.39 | 2.88 | — | |
| MInferenceBackbone Model=Qwen3-30B-A3B-Instruct-25072026.03 | 95.53 | 95.12 | 94.87 | 92.62 | 88.63 | 82.16 | 91.48 | 0.11 | 0.16 | 0.45 | 0.85 | 1.39 | 2.73 | — | |
| FlashPrefillBackbone Model=Qwen2.5-7B-Instruct2026.03 | 95.37 | 93.64 | 91.98 | 88.25 | 70.45 | 34.1 | 78.97 | 1.72 | — | 4.51 | 6.02 | 10.68 | 16.87 | — | |
| FlexPrefillBackbone Model=Qwen2.5-7B-Instruct2026.03 | 95.27 | 93.21 | 91.81 | 89.12 | 70.06 | 26.33 | 77.63 | 0.12 | 0.34 | 0.95 | 2.29 | 4 | 5.86 | — | |
| Full AttentionBackbone Model=Qwen2.5-7B-Instruct2026.03 | 95.21 | 93.17 | 92.18 | 90.14 | 71.87 | 25.23 | 77.97 | 1 | 1 | 1 | 1 | 1 | 1 | — | |
| XAttentionBackbone Model=Llama-3.1-8B-Instruct2026.03 | 95.01 | 92.16 | 91.91 | 90.06 | 85.17 | 71.68 | 87.67 | 0.77 | 1.27 | 1.83 | 2.34 | 3.19 | 3.48 | — | |
| XAttentionBackbone Model=Qwen3-30B-A3B-Instruct-25072026.03 | 94.72 | 93.68 | 92.32 | 90.44 | 88.12 | 83.69 | 90.5 | 0.84 | 1.43 | 2.02 | 2.45 | 2.97 | 3.45 | — | |
| XAttentionBackbone Model=Qwen2.5-7B-Instruct2026.03 | 94.52 | 92.14 | 89.72 | 86.21 | 66.65 | 26.36 | 75.93 | 0.84 | 1.56 | 2.32 | 3.04 | 3.64 | 4.34 | — | |
| BaselineModel=Gemma-3-12B, Attention Type=Sliding Window/Full Attention Hybrid, Batch Size=12026.05 | 94.01 | 89.12 | 85.98 | 80.76 | 68.89 | 61.22 | 79.99 | 1 | 1 | 1 | 1 | 1 | 1 | — | |
| ProxyAttnModel=Gemma-3-12B, Attention Type=Sliding Window/Full Attention Hybrid, Batch Size=12026.05 | 93.67 | 89.52 | 85.31 | 78.97 | 65.31 | 59.93 | 78.79 | 0.98 | 0.99 | 1.01 | 1.01 | 1.03 | 1.06 | — | |
| FlexPrefillModel=Gemma-3-12B, Attention Type=Sliding Window/Full Attention Hybrid, Batch Size=12026.05 | 93.63 | 89.16 | 85.49 | 79.23 | 65.69 | 58.63 | 78.64 | 0.98 | 0.99 | 1 | 1.01 | 1.02 | 1.04 | — | |
| MInferenceModel=Gemma-3-12B, Attention Type=Sliding Window/Full Attention Hybrid, Batch Size=12026.05 | 93.56 | 89.51 | 86.01 | 80.04 | 67.09 | 59.31 | 79.25 | 0.98 | 0.99 | 1 | 1 | 1.01 | 1.03 | — | |
| UniPrefillModel=Gemma-3-12B, Attention Type=Sliding Window/Full Attention Hybrid, Batch Size=12026.05 | 93.18 | 89.76 | 86.47 | 79.08 | 66.32 | 58.38 | 78.87 | 1.15 | 1.21 | 1.22 | 1.26 | 1.31 | 1.49 | — | |
| FlashMoBABackbone Model=Llama-3.1-8B-Instruct2026.03 | 93.14 | 83.23 | 79.28 | 72.13 | 60.12 | 31.26 | 69.86 | 0.75 | 1.16 | 1.99 | 3.34 | 4.7 | 5.98 | — | |
| XAttentionModel=Gemma-3-12B, Attention Type=Sliding Window/Full Attention Hybrid, Batch Size=12026.05 | 93.06 | 89.24 | 85.67 | 79.14 | 66.18 | 56.24 | 78.26 | 0.99 | 1 | 1.01 | 1.01 | 1.02 | 1.02 | — | |
| FlashMoBABackbone Model=Qwen2.5-7B-Instruct2026.03 | 92.36 | 83.68 | 76.11 | 68.12 | 49.31 | 20.13 | 64.95 | 0.73 | 1.14 | 1.97 | 3.34 | 4.7 | 5.98 | — | |
| FlashMoBABackbone Model=Qwen3-30B-A3B-Instruct-25072026.03 | 92.18 | 89.13 | 84.22 | 76.13 | 67.12 | 60.12 | 78.15 | 0.7 | 1.13 | 1.97 | 3.34 | 4.7 | 5.98 | — | |
| DHSAToken Density=12.5%, Sparsity Setting=Matched2025.10 | 92.1 | 88.5 | 88.7 | 76.2 | — | — | — | — | — | — | — | — | — | 71.5 | |
| DuoAttentionToken Density=12.5%, Sparsity Setting=Matched2025.10 | 92 | 89.3 | 89.7 | 72.9 | — | — | — | — | — | — | — | — | — | 69.1 | |
| MInferenceToken Density=12.5%, Sparsity Setting=Matched2025.10 | 91.7 | 88.4 | 88.6 | 74.7 | — | — | — | — | — | — | — | — | — | 64.2 | |
| SlimInferModel=Llama-3.1-8B-Instruct, Attention Type=Full Attention, Batch Size=12026.05 | 90.23 | 82.04 | 71.39 | 67.12 | 57.1 | 45.36 | 68.87 | 1.07 | 1.16 | 1.25 | 1.66 | 1.98 | 2.07 | — | |
| LazyLLMModel=Llama-3.1-8B-Instruct, Attention Type=Full Attention, Batch Size=12026.05 | 89.16 | 81.12 | 70.32 | 64.39 | 56.28 | 49.71 | 68.5 | 1.09 | 1.19 | 1.28 | 1.74 | 2.21 | 2.51 | — | |
| LazyLLMModel=Qwen3-Next-80B-A3B, Attention Type=Linear/Full Attention Hybrid, Batch Size=12026.05 | 89.13 | 82.37 | 70.69 | 64.37 | 58.12 | 55.17 | 69.98 | 1.11 | 1.18 | 1.29 | 1.4 | 1.55 | 1.74 | — | |
| SlimInferModel=Gemma-3-12B, Attention Type=Sliding Window/Full Attention Hybrid, Batch Size=12026.05 | 88.23 | 83.14 | 79.12 | 69.33 | 53.02 | 40.11 | 68.83 | 1.15 | 1.24 | 1.32 | 1.39 | 1.45 | 1.52 | — | |
| SlimInferModel=Qwen3-Next-80B-A3B, Attention Type=Linear/Full Attention Hybrid, Batch Size=12026.05 | 88.11 | 80.36 | 67.13 | 63.22 | 57.13 | 55.36 | 68.55 | 1.14 | 1.22 | 1.27 | 1.34 | 1.42 | 1.56 | — | |
| QuestToken Density=12.5%, Sparsity Setting=Matched2025.10 | 87.7 | 86 | 87.2 | 73.9 | — | — | — | — | — | — | — | — | — | 68.2 | |
| SeerAttentionToken Density=12.5%, Sparsity Setting=Matched2025.10 | 86.9 | 89 | 89.1 | 75 | — | — | — | — | — | — | — | — | — | 53.6 | |
| LazyLLMModel=Gemma-3-12B, Attention Type=Sliding Window/Full Attention Hybrid, Batch Size=12026.05 | 86.11 | 80.34 | 75.23 | 68.42 | 54.12 | 43.38 | 67.93 | 1.23 | 1.32 | 1.37 | 1.42 | 1.53 | 1.64 | — | |
| BlockSparseToken Density=12.5%, Sparsity Setting=Matched2025.10 | 60.3 | 65.4 | 69.3 | 53 | — | — | — | — | — | — | — | — | — | 43.1 |