Long-context Understanding on LongBench (Selected Subtasks)
46.25Average ScoreLKV
Evaluation results
| Method | Links | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| LKVModel=Qwen3-8B, Retention Ratio (R)=0.52026.04 | 46.25 | — | — | — | — | — | — | — | — | |
| LKVModel=Qwen3-8B, Retention Ratio (R)=0.32026.04 | 46.01 | — | — | — | — | — | — | — | — | |
| LKVModel=Qwen3-8B, Retention Ratio (R)=0.22026.04 | 45.93 | — | — | — | — | — | — | — | — | |
| LKVModel=Qwen3-8B, Retention Ratio (R)=0.42026.04 | 45.86 | — | — | — | — | — | — | — | — | |
| LKVModel=Qwen3-8B, Retention Ratio (R)=0.152026.04 | 45.41 | — | — | — | — | — | — | — | — | |
| Ada-SnapKVModel=Qwen3-8B, Retention Ratio (R)=0.52026.04 | 44.53 | — | — | — | — | — | — | — | — | |
| SnapKVModel=Qwen3-8B, Retention Ratio (R)=0.52026.04 | 44.41 | — | — | — | — | — | — | — | — | |
| Ada-SnapKVModel=Qwen3-8B, Retention Ratio (R)=0.42026.04 | 43.53 | — | — | — | — | — | — | — | — | |
| SnapKVModel=Qwen3-8B, Retention Ratio (R)=0.42026.04 | 43.2 | — | — | — | — | — | — | — | — | |
| LKVModel=Qwen3-8B, Retention Ratio (R)=0.12026.04 | 43.17 | — | — | — | — | — | — | — | — | |
| Ada-SnapKVModel=Qwen3-8B, Retention Ratio (R)=0.32026.04 | 42.44 | — | — | — | — | — | — | — | — | |
| SnapKVModel=Qwen3-8B, Retention Ratio (R)=0.32026.04 | 41.41 | — | — | — | — | — | — | — | — | |
| Ada-SnapKVModel=Qwen3-8B, Retention Ratio (R)=0.22026.04 | 40.21 | — | — | — | — | — | — | — | — | |
| PyramidKVModel=Qwen3-8B, Retention Ratio (R)=0.52026.04 | 40.2 | — | — | — | — | — | — | — | — | |
| PyramidKVModel=Qwen3-8B, Retention Ratio (R)=0.42026.04 | 39.82 | — | — | — | — | — | — | — | — | |
| SnapKVModel=Qwen3-8B, Retention Ratio (R)=0.22026.04 | 39.16 | — | — | — | — | — | — | — | — | |
| PyramidKVModel=Qwen3-8B, Retention Ratio (R)=0.32026.04 | 38.23 | — | — | — | — | — | — | — | — | |
| Ada-SnapKVModel=Qwen3-8B, Retention Ratio (R)=0.152026.04 | 37.9 | — | — | — | — | — | — | — | — | |
| SnapKVModel=Qwen3-8B, Retention Ratio (R)=0.152026.04 | 36.72 | — | — | — | — | — | — | — | — | |
| PyramidKVModel=Qwen3-8B, Retention Ratio (R)=0.22026.04 | 36.19 | — | — | — | — | — | — | — | — | |
| Ada-SnapKVModel=Qwen3-8B, Retention Ratio (R)=0.12026.04 | 34.1 | — | — | — | — | — | — | — | — | |
| PyramidKVModel=Qwen3-8B, Retention Ratio (R)=0.152026.04 | 34.01 | — | — | — | — | — | — | — | — | |
| SnapKVModel=Qwen3-8B, Retention Ratio (R)=0.12026.04 | 33.31 | — | — | — | — | — | — | — | — | |
| DuoAttentionModel=Qwen3-8B, Retention Ratio (R)=0.52026.04 | 32.66 | — | — | — | — | — | — | — | — | |
| PyramidKVModel=Qwen3-8B, Retention Ratio (R)=0.12026.04 | 32.29 | — | — | — | — | — | — | — | — | |
| Hybrid Gated DeltaNet + M2RNN-5Model Scale=7B MoE, Active Parameters=1B active2026.03 | 31.2 | 12.2 | 16 | 16.1 | 59 | 73.5 | 36 | 18 | 18.9 | |
| DuoAttentionModel=Qwen3-8B, Retention Ratio (R)=0.42026.04 | 30.67 | — | — | — | — | — | — | — | — | |
| Hybrid Gated DeltaNet + M2RNN-1Model Scale=7B MoE, Active Parameters=1B active2026.03 | 30.4 | 17.2 | 16.5 | 6 | 69.5 | 64.9 | 34.1 | 20.1 | 15.1 | |
| DuoAttentionModel=Qwen3-8B, Retention Ratio (R)=0.32026.04 | 28.89 | — | — | — | — | — | — | — | — | |
| DuoAttentionModel=Qwen3-8B, Retention Ratio (R)=0.22026.04 | 27.4 | — | — | — | — | — | — | — | — | |
| DuoAttentionModel=Qwen3-8B, Retention Ratio (R)=0.12026.04 | 27.13 | — | — | — | — | — | — | — | — | |
| DuoAttentionModel=Qwen3-8B, Retention Ratio (R)=0.152026.04 | 26.85 | — | — | — | — | — | — | — | — | |
| Gated DeltaNetModel Scale=7B MoE, Active Parameters=1B active2026.03 | 25 | 4.8 | 20.3 | 12.2 | 47 | 61.6 | 36.2 | 9.7 | 7.9 | |
| Hybrid M2RNNModel Scale=7B MoE, Active Parameters=1B active2026.03 | 24.6 | 10.9 | 13.9 | 21.1 | 52 | 44.7 | 18.5 | 19.8 | 15.7 | |
| M2RNNModel Scale=7B MoE, Active Parameters=1B active2026.03 | 23.4 | 8.1 | 19.1 | 11.8 | 26.5 | 51.3 | 30.7 | 20.4 | 19.3 | |
| Hybrid Gated DeltaNetModel Scale=7B MoE, Active Parameters=1B active2026.03 | 23.1 | 10.4 | 15.8 | 16.7 | 57.5 | 35.1 | 23.6 | 9.7 | 16.3 | |
| Hybrid Mamba-2 + M2RNN-5Model Scale=7B MoE, Active Parameters=1B active2026.03 | 22.7 | 18.6 | 10.3 | 7.1 | 60.5 | 45.7 | 21.4 | 9.7 | 8 | |
| Mamba-2Model Scale=7B MoE, Active Parameters=1B active2026.03 | 22.5 | 5.1 | 16 | 13.4 | 42 | 59.7 | 22 | 13.4 | 8.7 | |
| GRUModel Scale=7B MoE, Active Parameters=1B active2026.03 | 19.4 | 5.7 | 18 | 10.8 | 9 | 52.4 | 24.6 | 20.9 | 13.9 | |
| Hybrid Mamba-2Model Scale=7B MoE, Active Parameters=1B active2026.03 | 18.8 | 9.1 | 7.7 | 20.3 | 62.5 | 23.3 | 12.1 | 5.6 | 9.8 | |
| RNNModel Scale=7B MoE, Active Parameters=1B active2026.03 | 14.2 | 9.3 | 16.7 | 10.5 | 4 | 27.5 | 13 | 17.1 | 15.8 | |
| Hybrid Mamba-2 + M2RNN-1Model Scale=7B MoE, Active Parameters=1B active2026.03 | 13.1 | 8.6 | 3.9 | 21.2 | 25.5 | 15 | 11.5 | 7.5 | 11.6 | |
| Transformer++Model Scale=7B MoE, Active Parameters=1B active2026.03 | 11.2 | 6.1 | 6.1 | 17 | 16 | 11.4 | 10 | 13.1 | 10.2 |