Long-context Understanding on HELMET 2025
61.44Accuracy (8K Context)MInference
Evaluation results
| Method | Links | |||||||
|---|---|---|---|---|---|---|---|---|
| MInferenceModel (Backbone)=Llama-3.1-8B-Instruct2025.12 | 61.44 | 56.34 | 57.96 | 55.49 | 53.69 | 48.29 | 55.37 | |
| UniSparseModel (Backbone)=Llama-3.1-8B-Instruct, Sparsity Threshold (Top-P)=0.952025.12 | 61.37 | 48.65 | 58.34 | 56.22 | 55.27 | 50.22 | 56.21 | |
| FlashAttentionModel (Backbone)=Llama-3.1-8B-Instruct2025.12 | 61.06 | — | 58.73 | 56.57 | 55.92 | 49.79 | 56.41 | |
| UniSparseModel (Backbone)=Llama-3.1-8B-Instruct, Sparsity Threshold (Top-P)=0.92025.12 | 60.86 | 61.82 | 57.7 | 56.21 | 54.24 | 48.91 | 55.58 | |
| XAttentionModel (Backbone)=Llama-3.1-8B-Instruct, Sparsity Threshold (Top-P)=0.952025.12 | 60.76 | 47.45 | 57.8 | 55.92 | 54.75 | 49.37 | 55.72 | |
| FlexPrefillModel (Backbone)=Llama-3.1-8B-Instruct, Sparsity Threshold (Top-P)=0.992025.12 | 60.66 | 38.52 | 58.05 | 56.22 | 55.44 | 49.68 | 56.01 | |
| XAttentionModel (Backbone)=Llama-3.1-8B-Instruct, Sparsity Threshold (Top-P)=0.92025.12 | 59.81 | 59.8 | 57.4 | 55.55 | 53.74 | 47.74 | 54.85 | |
| FlexPrefillModel (Backbone)=Llama-3.1-8B-Instruct, Sparsity Threshold (Top-P)=0.952025.12 | 55.89 | 65.92 | 54.73 | 53.14 | 50.45 | 48.42 | 52.53 | |
| FlashAttentionModel (Backbone)=Qwen2.5-7B-Instruct2025.12 | 55.1 | — | 50.54 | 48.51 | 44.85 | 39.33 | 47.67 | |
| MInferenceModel (Backbone)=Qwen2.5-7B-Instruct2025.12 | 54.91 | 46.95 | 50.36 | 46.93 | 42.62 | 34.05 | 45.77 | |
| XAttentionModel (Backbone)=Qwen2.5-7B-Instruct, Sparsity Threshold (Top-P)=0.952025.12 | 53.93 | 47.31 | 49.59 | 47.1 | 44.65 | 38.19 | 46.69 | |
| UniSparseModel (Backbone)=Qwen2.5-7B-Instruct, Sparsity Threshold (Top-P)=0.952025.12 | 53.91 | 47.51 | 49.96 | 47.76 | 44.95 | 38.58 | 47.03 | |
| UniSparseModel (Backbone)=Qwen2.5-7B-Instruct, Sparsity Threshold (Top-P)=0.92025.12 | 53.13 | 60.84 | 48.62 | 46.93 | 44.22 | 38.11 | 46.2 | |
| FlexPrefillModel (Backbone)=Qwen2.5-7B-Instruct, Sparsity Threshold (Top-P)=0.992025.12 | 52.77 | 37.75 | 48.91 | 46.39 | 43.06 | 36.6 | 45.55 | |
| XAttentionModel (Backbone)=Qwen2.5-7B-Instruct, Sparsity Threshold (Top-P)=0.92025.12 | 52.46 | 59.08 | 47.99 | 45.63 | 43.17 | 36.92 | 45.23 | |
| FlexPrefillModel (Backbone)=Qwen2.5-7B-Instruct, Sparsity Threshold (Top-P)=0.952025.12 | 44.23 | 63.03 | 40.79 | 39.87 | 36.28 | 30.06 | 38.25 |