Long-context Language Understanding on LongBench (7-Category Summary including QA, Code, and Synthetic)
48.8Single-Doc QA AccuracyFull
Evaluation Results
| Method | Links | |||||||
|---|---|---|---|---|---|---|---|---|
| FullBackbone=Llama-3.1-8B2025.10 | 48.8 | 41.8 | 17.79 | 29.73 | 24.77 | 66.82 | 38.28 | |
| XAttentionBackbone=Llama-3.1-8B2025.10 | 48.26 | 40.23 | 17.86 | 31.35 | 26.19 | 54.64 | 36.42 | |
| PBS-AttnBackbone=Llama-3.1-8B2025.10 | 48 | 42.09 | 17.72 | 28.36 | 24.25 | 63.8 | 37.37 | |
| MInferenceBackbone=Llama-3.1-8B2025.10 | 47.21 | 40.93 | 17.72 | 29.36 | 24.77 | 62.36 | 37.06 | |
| FlexPrefillBackbone=Llama-3.1-8B2025.10 | 47.03 | 38.57 | 17.78 | 30.38 | 24.88 | 24.71 | 30.56 | |
| MeanPoolingBackbone=Llama-3.1-8B2025.10 | 46.61 | 40.66 | 17.85 | 30.64 | 26.1 | 58.14 | 36.67 | |
| FullBackbone=Qwen-2.5-7B-1M2025.10 | 44.21 | 42.97 | 16.01 | 47.48 | 3.91 | 67.5 | 37.01 | |
| XAttentionBackbone=Qwen-2.5-7B-1M2025.10 | 43.82 | 42.21 | 16.07 | 48.3 | 3.83 | 63.33 | 36.26 | |
| PBS-AttnBackbone=Qwen-2.5-7B-1M2025.10 | 43.01 | 41.4 | 16.12 | 47.36 | 4 | 66.33 | 36.37 | |
| MInferenceBackbone=Qwen-2.5-7B-1M2025.10 | 42.82 | 41.76 | 16.01 | 46.41 | 3.8 | 66.5 | 36.21 | |
| MeanPoolingBackbone=Qwen-2.5-7B-1M2025.10 | 39.39 | 40.96 | 15.95 | 49.07 | 4.8 | 40.83 | 31.83 | |
| FlexPrefillBackbone=Qwen-2.5-7B-1M2025.10 | 38.44 | 37.51 | 15.87 | 46.12 | 6.46 | 26.67 | 28.51 |