Long-context modeling on RULER (test)
96.74Accuracy (4K Context)Full Attention
Evaluation results
| Method | Links | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| Full AttentionBase Model=LLaMA3.1-8B-Instruct2025.12 | 96.74 | 94.03 | 92.02 | 84.17 | 81.32 | 76.89 | 87.52 | 1,282.05 | |
| MInferenceBase Model=LLaMA3.1-8B-Instruct2025.12 | 96.54 | 94.06 | 91.37 | 85.79 | 83.03 | 54.13 | 84.15 | 837.09 | |
| TCA-AttentionBase Model=LLaMA3.1-8B-Instruct2025.12 | 96.31 | 95.38 | 93.92 | 86.38 | 82.89 | 77.46 | 88.72 | 453.22 | |
| XAttentionBase Model=LLaMA3.1-8B-Instruct2025.12 | 96.15 | 93.95 | 93.71 | 90.9 | 83.35 | 72.57 | 88.44 | 504 | |
| MInferenceBase Model=Qwen2.5-7B-Instruct2025.12 | 96.08 | 94.92 | 91.69 | 89.92 | 70.46 | 51.62 | 82.45 | 771.61 | |
| Full AttentionBase Model=Qwen2.5-7B-Instruct2025.12 | 96 | 94.85 | 91.77 | 89.85 | 70.38 | 52.92 | 82.63 | 1,101.01 | |
| TCA-AttentionBase Model=Qwen2.5-7B-Instruct2025.12 | 96 | 94.77 | 91.92 | 90.08 | 69.85 | 52.31 | 82.49 | 396.72 | |
| FlexPrefillBase Model=LLaMA3.1-8B-Instruct2025.12 | 95.99 | 93.67 | 92.73 | 88.14 | 81.14 | 74.67 | 87.72 | 1,021.47 | |
| FlexPrefillBase Model=Qwen2.5-7B-Instruct2025.12 | 95.62 | 94.31 | 92 | 88.23 | 70.23 | 52.15 | 82.09 | 892.55 | |
| XAttentionBase Model=Qwen2.5-7B-Instruct2025.12 | 95.45 | 92.91 | 92.04 | 88.84 | 68.84 | 55.36 | 82.24 | 441.36 |