Long-context language understanding on LongBench (QQA, MFQ, HQA, 2WM, Mus, TRC, TQA subset)
25.7QQAMamba2-8B
Evaluation Results
| Method | Links | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| Mamba2-8BTokens (B)=35002025.05 | 25.7 | 29.3 | 30 | 24.6 | 11.1 | 54 | 77.77 | 36.1 | |
| LoLAη (sliding window size)=512, λ (global cache size)=10242025.05 | 23.1 | 30.8 | 11.8 | 17.7 | 3.4 | 57.5 | 50.2 | 27.8 | |
| LoLCATs+η (sliding window size)=1536, λ (global cache size)=02025.05 | 1.8 | 6.1 | 4.4 | 5.6 | 1.5 | 37.5 | 5.8 | 9 | |
| LoLCATs-8BTokens (B)=0.042025.05 | 1.1 | 2.5 | 1 | 2.1 | 0.7 | 8.2 | 1 | 2.4 |