Long-context Language Modeling on LongBench 16K context length
13.3NrtvQA ScoreUltraLLaDA + BA-Att
Evaluation results
| Method | Links | ||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| UltraLLaDA + BA-AttAttention=BA-Att, Sparsity=> 50%2026.05 | 13.3 | 13.6 | 27.9 | 20.3 | 13.8 | 9.8 | 8.7 | 12.6 | 20.8 | 7.9 | 12.7 | 8.3 | 75.5 | 90.9 | 35.5 | 34.4 | 1.3 | 82.3 | 70.9 | 69 | 63.1 | 37.2 | |
| UltraLLaDAAttention=Dense Attention2026.05 | 12.1 | 14.5 | 27.4 | 19.9 | 13.4 | 11 | 9.7 | 20.4 | 20.8 | 7.8 | 14.7 | 7.5 | 79.5 | 92 | 36.2 | 40.5 | 0.7 | 81.3 | 72.9 | 66.2 | 56.9 | 37.2 | |
| LLaDA1.5 + BA-AttAttention=BA-Att, Sparsity=> 50%2026.05 | 4.9 | 20.4 | 21.8 | 20.1 | 6.3 | 8 | 3.7 | 11.2 | 17.4 | 5.4 | 14.4 | 6.7 | 71 | 60.1 | 33.6 | 22.3 | 2.1 | 28.4 | 68.2 | 67.9 | 60.8 | 31.3 | |
| LLaDA1.5 + XAttAttention=XAtt2026.05 | 4.3 | 20.3 | 21.5 | 19.9 | 7.5 | 7.9 | 3.5 | 10.8 | 16.8 | 5.8 | 14.2 | 7.5 | 61.5 | 61.5 | 28.7 | 19.3 | 1.9 | 27.7 | 56.8 | 61.9 | 55 | 28.8 | |
| LLaDA1.5Attention=Dense Attention2026.05 | 4.1 | 20.4 | 21.8 | 20.8 | 6.3 | 8.1 | 3.4 | 10.5 | 17.3 | 5.4 | 14.3 | 6.7 | 73.5 | 63.6 | 32.2 | 21.5 | 1.9 | 29.5 | 70.9 | 68 | 60.6 | 31.5 | |
| UltraLLaDA + XAttAttention=XAtt2026.05 | 4.1 | 13.9 | 25.2 | 20.1 | 6.2 | 6.8 | 5.2 | 12.9 | 19.9 | 7 | 16 | 7.4 | 75 | 85.2 | 33.3 | 28.5 | 2.1 | 81.4 | 64.6 | 64.5 | 63.5 | 34.9 |