Long-context language modeling evaluation on HELMET
0Average SparsityFullAttention
Evaluation results
| Method | Links | |||||||
|---|---|---|---|---|---|---|---|---|
| FullAttentionBackbone=Yi-9B-200K, Sparsity Parameter=-2026.02 | 0 | 50.33 | 48.87 | 46.11 | 43.83 | 41.07 | 46.04 | |
| FullAttentionBackbone=Qwen3-30B-A3B-Instruct, Sparsity Parameter=-2026.02 | 0 | 67.09 | 67.42 | 66.78 | 65.73 | 62.8 | 65.96 | |
| FullAttentionBackbone=Llama2026.02 | 0 | 61.17 | 58.67 | 56.71 | 55.81 | 49.74 | 56.42 | |
| FullAttentionBackbone=Qwen2026.02 | 0 | 54.75 | 50.66 | 48.4 | 44.98 | 39.59 | 47.68 | |
| FlexPrefillBackbone=Qwen, Sparsity Threshold=0.992026.02 | 37.88 | 53.18 | 49.05 | 46.48 | 43.01 | 35.52 | 45.45 | |
| FlexPrefillBackbone=Llama, Sparsity Threshold=0.992026.02 | 38.63 | 60.73 | 58.15 | 56.22 | 55.15 | 49.87 | 56.02 | |
| FlexPrefillBackbone=Yi-9B-200K, Sparsity Parameter=0.992026.02 | 40.16 | 50.87 | 50.01 | 46 | 43.87 | 40.92 | 46.33 | |
| XAttentionBackbone=Qwen3-30B-A3B-Instruct, Sparsity Parameter=0.952026.02 | 41.96 | 66.51 | 67.22 | 66.38 | 63.75 | 61.17 | 65.01 | |
| RRAttentionBackbone=Qwen3-30B-A3B-Instruct, Sparsity Parameter=0.952026.02 | 43.01 | 66.96 | 67.63 | 66.55 | 64.46 | 61.66 | 65.45 | |
| XAttentionBackbone=Qwen, Sparsity Threshold=0.952026.02 | 47.35 | 53.9 | 49.77 | 46.96 | 44.49 | 38.09 | 46.64 | |
| XAttentionBackbone=Llama, Sparsity Threshold=0.952026.02 | 47.5 | 60.79 | 58.1 | 55.57 | 54.78 | 49.45 | 55.74 | |
| RRAttentionBackbone=Qwen, Sparsity Threshold=0.952026.02 | 47.56 | 54.22 | 50.02 | 47.92 | 45.17 | 38.51 | 47.17 | |
| FlexPrefillBackbone=Qwen3-30B-A3B-Instruct, Sparsity Parameter=0.972026.02 | 48.22 | 66.15 | 65.61 | 64.97 | 63.2 | 60.8 | 64.15 | |
| RRAttentionBackbone=Llama, Sparsity Threshold=0.952026.02 | 48.68 | 61.22 | 58.21 | 56.07 | 55.33 | 50.37 | 56.24 | |
| XAttentionBackbone=Yi-9B-200K, Sparsity Parameter=0.952026.02 | 50.8 | 50.41 | 48.51 | 44.73 | 43.69 | 41.17 | 45.7 | |
| RRAttentionBackbone=Yi-9B-200K, Sparsity Parameter=0.952026.02 | 51.86 | 50.95 | 49.45 | 47.13 | 44.12 | 41.06 | 46.54 | |
| XAttentionBackbone=Qwen3-30B-A3B-Instruct, Sparsity Parameter=0.902026.02 | 55.2 | 66.78 | 67.19 | 65.25 | 63.12 | 60.04 | 64.48 | |
| RRAttentionBackbone=Qwen3-30B-A3B-Instruct, Sparsity Parameter=0.902026.02 | 56.26 | 67.01 | 66.6 | 65.71 | 62.97 | 60.7 | 64.6 | |
| FlexPrefillBackbone=Qwen3-30B-A3B-Instruct, Sparsity Parameter=0.952026.02 | 58.42 | 64.08 | 64.11 | 63.15 | 61.64 | 59.3 | 62.46 | |
| XAttentionBackbone=Qwen, Sparsity Threshold=0.902026.02 | 59.15 | 52.53 | 47.72 | 45.72 | 43.35 | 37.04 | 45.27 | |
| RRAttentionBackbone=Qwen, Sparsity Threshold=0.902026.02 | 59.53 | 53.37 | 49.31 | 47.35 | 44.62 | 38.48 | 46.63 | |
| XAttentionBackbone=Llama, Sparsity Threshold=0.902026.02 | 59.88 | 59.92 | 57.15 | 55.54 | 53.66 | 47.82 | 54.82 | |
| RRAttentionBackbone=Llama, Sparsity Threshold=0.902026.02 | 61.02 | 60.95 | 58.43 | 55.86 | 54.27 | 48.85 | 55.67 | |
| FlexPrefillBackbone=Qwen, Sparsity Threshold=0.952026.02 | 63.21 | 44.11 | 40.59 | 40.6 | 36.39 | 29.66 | 38.27 | |
| XAttentionBackbone=Yi-9B-200K, Sparsity Parameter=0.902026.02 | 63.85 | 50.28 | 47.79 | 44.17 | 43.31 | 40.19 | 45.15 | |
| RRAttentionBackbone=Yi-9B-200K, Sparsity Parameter=0.902026.02 | 64.05 | 50.79 | 48.58 | 46.48 | 43.89 | 40.77 | 46.1 | |
| FlexPrefillBackbone=Yi-9B-200K, Sparsity Parameter=0.952026.02 | 65.98 | 44.99 | 44.51 | 43.27 | 41.31 | 38.73 | 42.56 | |
| FlexPrefillBackbone=Llama, Sparsity Threshold=0.952026.02 | 66.07 | 56.02 | 54.48 | 53.01 | 50.77 | 48.42 | 52.54 |