Long-context understanding on RULER (test) (4K-128K Contexts)
96.6Accuracy (4K Context)Llama-3.1-8B-Instruct (Dense)
Evaluation results
| Method | Links | |||||||
|---|---|---|---|---|---|---|---|---|
| Llama-3.1-8B-Instruct (Dense)Model=Llama-3.1-8B-Instruct, Sparsity=Dense2025.07 | 96.6 | 95.3 | 94.8 | 91.3 | 86.3 | 78.1 | 90.4 | |
| XAttentionModel=Llama-3.1-8B-Instruct2025.07 | 96.5 | 94.9 | 94.5 | 91.8 | 85.7 | 73.8 | 89.5 | |
| TriangleMixModel=Llama-3.1-8B-Instruct2025.07 | 96.3 | 95.1 | 94.7 | 91.3 | 86.3 | 77.5 | 90.2 | |
| MInferenceModel=Llama-3.1-8B-Instruct2025.07 | 96.3 | 95.1 | 95 | 90.5 | 86.8 | 75 | 89.8 | |
| TriangleMix + MInferenceModel=Llama-3.1-8B-Instruct2025.07 | 96.2 | 95 | 94.5 | 90.8 | 87.2 | 75.8 | 89.9 | |
| TriangleMix + XAttentionModel=Llama-3.1-8B-Instruct2025.07 | 96.2 | 95.1 | 94.9 | 91.9 | 85.7 | 72.9 | 89.4 | |
| Qwen2.5-7B-Instruct (Dense)Model=Qwen2.5-7B-Instruct, Sparsity=Dense2025.07 | 95.8 | 93.6 | 92.6 | 84.5 | 81.9 | 67.4 | 86 | |
| TriangleMix + MInferenceModel=Qwen2.5-7B-Instruct2025.07 | 95.7 | 93.5 | 92.4 | 84.8 | 76.9 | 63.5 | 84.5 | |
| MInferenceModel=Qwen2.5-7B-Instruct2025.07 | 95.6 | 93.8 | 92.8 | 84.9 | 79.1 | 68 | 85.7 | |
| TriangleMixModel=Qwen2.5-7B-Instruct2025.07 | 95.5 | 93.8 | 92.1 | 84.6 | 80.2 | 67.7 | 85.6 | |
| XAttentionModel=Qwen2.5-7B-Instruct2025.07 | 95.4 | 93.7 | 92 | 82.9 | 77.8 | 66.2 | 84.7 | |
| TriangleMix + XAttentionModel=Qwen2.5-7B-Instruct2025.07 | 95.3 | 93.3 | 91.5 | 82.2 | 77.4 | 64.9 | 84.1 | |
| TriangleMix + FlexPrefillModel=Llama-3.1-8B-Instruct2025.07 | 95.2 | 95 | 94.7 | 92.8 | 86.8 | 76.2 | 90.1 | |
| FlexPrefillModel=Llama-3.1-8B-Instruct2025.07 | 95 | 94.7 | 94.5 | 92.8 | 86.5 | 75.9 | 89.9 | |
| XAttentionModel=Llama-3-8B-Instruct-262K2025.07 | 93.6 | 91.1 | 87.7 | 84.8 | 82.6 | 78.9 | 86.5 | |
| TriangleMixModel=Llama-3-8B-Instruct-262K2025.07 | 93.5 | 91 | 88.1 | 85 | 82.4 | 79.6 | 86.6 | |
| Llama-3-8B-Instruct-262K (Dense)Model=Llama-3-8B-Instruct-262K, Sparsity=Dense2025.07 | 93.4 | 90.3 | 88.8 | 85.1 | 82.2 | 79.4 | 86.5 | |
| MInferenceModel=Llama-3-8B-Instruct-262K2025.07 | 93.4 | 90.7 | 88.8 | 85.1 | 82.6 | 80.1 | 86.8 | |
| TriangleMix + XAttentionModel=Llama-3-8B-Instruct-262K2025.07 | 93.3 | 91.2 | 87.9 | 84.3 | 82.5 | 79.3 | 86.4 | |
| TriangleMix + MInferenceModel=Llama-3-8B-Instruct-262K2025.07 | 93 | 90.4 | 88.6 | 84.4 | 82.7 | 80.1 | 86.5 | |
| TriangleMix + FlexPrefillModel=Qwen2.5-7B-Instruct2025.07 | 92.2 | 91.1 | 89.5 | 77 | 73.3 | 52.7 | 79.3 | |
| FlexPrefillModel=Qwen2.5-7B-Instruct2025.07 | 91.1 | 89.7 | 88.5 | 75.4 | 72.5 | 51.2 | 78.1 | |
| FlexPrefillModel=Llama-3-8B-Instruct-262K2025.07 | 90.3 | 87.7 | 88 | 83.8 | 80.2 | 75.7 | 84.3 | |
| TriangleMix + FlexPrefillModel=Llama-3-8B-Instruct-262K2025.07 | 90.2 | 87.3 | 87.3 | 83.8 | 80 | 76.6 | 84.2 |