Long-context Language Modeling on RULER (test)
80SparsityProxyAttn
Evaluation Results
| Method | Links | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| ProxyAttnBackbone=Llama3.1-8B-Instruct, gamma=0.902025.09 | 80 | 95.76 | 94.32 | 93.52 | 91.05 | 86.14 | 75.75 | 89.42 | 86.31 | |
| SeerAttentionBackbone=Llama3.1-8B-Instruct, additional training=true2025.09 | 77 | 96.2 | 94.44 | 93.8 | 90.01 | 84.99 | 74.34 | 88.96 | 85.6 | |
| ProxyAttnBackbone=Qwen2.5-7B-Instruct-1M, gamma=0.902025.09 | 74 | 94.57 | 91.34 | 89.19 | 86.55 | 83.76 | 76.8 | 87.04 | 84.32 | |
| FlexPrefillBackbone=Llama3.1-8B-Instruct2025.09 | 72 | 96.03 | 94.07 | 94.35 | 92.05 | 85.21 | 75.12 | 89.47 | 86.28 | |
| MInferenceBackbone=Llama3.1-8B-Instruct2025.09 | 70 | 96.23 | 94.22 | 93.92 | 88.82 | 84.64 | 70.74 | 88.09 | 84.25 | |
| XAttentionBackbone=Llama3.1-8B-Instruct2025.09 | 69 | 96.19 | 94.27 | 93.99 | 91.04 | 85.15 | 72.64 | 88.88 | 85.35 | |
| ProxyAttnBackbone=Llama3.1-8B-Instruct, gamma=0.952025.09 | 69 | 96.07 | 94.57 | 94.01 | 91.55 | 86.63 | 78.25 | 90.18 | 87.43 | |
| FlexPrefillBackbone=Qwen2.5-7B-Instruct-1M2025.09 | 69 | 94.09 | 90.97 | 89.27 | 87.59 | 81.85 | 76.24 | 86.67 | 83.85 | |
| MInferenceBackbone=Qwen2.5-7B-Instruct-1M2025.09 | 63 | 94.41 | 91.43 | 89.18 | 87.56 | 82.73 | 76.6 | 86.99 | 84.2 | |
| XAttentionBackbone=Qwen2.5-7B-Instruct-1M2025.09 | 61 | 92.92 | 88.22 | 84.45 | 84.09 | 79.99 | 73.74 | 83.9 | 81.02 | |
| ProxyAttnBackbone=Qwen2.5-7B-Instruct-1M, gamma=0.952025.09 | 61 | 94.61 | 92.07 | 89.75 | 86.68 | 83.57 | 77.09 | 87.3 | 84.53 | |
| FullAttentionBackbone=Llama3.1-8B-Instruct2025.09 | 0 | 96.17 | 94.21 | 93.72 | 89.56 | 86.13 | 76.95 | 89.46 | 86.49 | |
| FullAttentionBackbone=Qwen2.5-7B-Instruct-1M2025.09 | 0 | 94.62 | 91.45 | 89.49 | 88.46 | 84.58 | 78.9 | 87.92 | 85.53 |