Document Understanding on LongBench
37.23CCXATTN
Evaluation Results
| Method | Links | |||||||
|---|---|---|---|---|---|---|---|---|
| XATTNBackbone=Llama-3.1-8B-Instruct2026.02 | 37.23 | 63.61 | 22.15 | 28.23 | 25.3 | 50.95 | 37.91 | |
| DenseBackbone=Llama-3.1-8B-Instruct2026.02 | 36.08 | 64.29 | 27.54 | 31.19 | 25.1 | 67.92 | 42.02 | |
| StemBackbone=Llama-3.1-8B-Instruct2026.02 | 35.86 | 62.89 | 26.33 | 30.53 | 24.93 | 68.32 | 41.48 | |
| MINFBackbone=Llama-3.1-8B-Instruct2026.02 | 35.43 | 62.98 | 25.42 | 29 | 25.12 | 68.41 | 41.06 | |
| FLEXBackbone=Llama-3.1-8B-Instruct2026.02 | 34.65 | 59.51 | 17.17 | 21.62 | 24.46 | 59.1 | 36.09 | |
| DSABackbone=DeepSeek-V3.2 (trained sparse)2026.02 | 32.42 | 41.85 | 51.83 | 48.3 | 22.28 | 60.35 | 42.84 | |
| DSA + StemBackbone=DeepSeek-V3.2 (trained sparse)2026.02 | 32.02 | 44.03 | 51.67 | 48.44 | 21.97 | 60.85 | 43.16 | |
| XATTNBackbone=Qwen3-8B2026.02 | 21.03 | 62.3 | 11.42 | 14.64 | 20.47 | 52.92 | 30.46 | |
| FLEXBackbone=Qwen3-8B2026.02 | 19.75 | 55.31 | 10.76 | 13.49 | 21.18 | 50.83 | 28.55 | |
| StemBackbone=Qwen3-8B2026.02 | 19.43 | 61.84 | 11.22 | 14.97 | 20.21 | 62.19 | 31.64 | |
| DenseBackbone=Qwen3-8B2026.02 | 19.09 | 63.1 | 11.23 | 15.06 | 20.63 | 62.92 | 32.01 | |
| MINFBackbone=Qwen3-8B2026.02 | 18.91 | 61.87 | 10.78 | 14.46 | 20.26 | 55.32 | 30.27 |