Language Modeling on Qwen2.5-7B
3.22PPLMLA
Evaluation Results
| Method | Links | ||||
|---|---|---|---|---|---|
| MLASequence Length=16K2026.03 | 3.22 | 4,456.9 | 12.8 | — | |
| FastMKASequence Length=16K2026.03 | 3.26 | 1,248.3 | 8.4 | — | |
| GQASequence Length=16K2026.03 | 3.28 | 5,012.4 | 18.6 | — | |
| MHASequence Length=16K2026.03 | 3.31 | 6,234.7 | 21.4 | — | |
| HyperQuantSetting=KV cache, rate=2 bps, residual_window=32-token2026.06 | — | — | — | 7.4 | |
| OCTOPUSSetting=KV cache, rate=2 bps, residual_window=32-token2026.06 | — | — | — | 34.7 |