Long-context language understanding on LongBench v2
46.32Overall AccuracyHyLRA
Evaluation results
| Method | Links | |||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| HyLRAModel=DeepSeek-R12026.01 | 46.32 | 49.48 | 44.37 | 52.78 | 43.72 | 40.74 | — | — | — | — | — | — | — | — | — | — | — | |
| Full AttentionModel=QWQ-32B-W8A82026.01 | 45.73 | 48.96 | 43.73 | 52.22 | 44.19 | 37.96 | — | — | — | — | — | — | — | — | — | — | — | |
| HyLRAModel=QWQ-32B-W8A82026.01 | 45.03 | 48.96 | 42.61 | 50 | 42.79 | 41.2 | — | — | — | — | — | — | — | — | — | — | — | |
| Full AttentionModel=DeepSeek-R12026.01 | 44.33 | 48.44 | 41.8 | 48.89 | 40 | 45.37 | — | — | — | — | — | — | — | — | — | — | — | |
| Jump3Model=DeepSeek-R12026.01 | 42.54 | 43.75 | 41.8 | 50 | 39.07 | 37.04 | — | — | — | — | — | — | — | — | — | — | — | |
| Jump3Model=QWQ-32B-W8A82026.01 | 40.76 | 43.75 | 38.91 | 45.56 | 38.14 | 37.96 | — | — | — | — | — | — | — | — | — | — | — | |
| FullBackbone=Qwen3-8B2026.02 | 34.19 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Qwen2.5-7BBase Model=Qwen2.5-7B2026.02 | 33.59 | — | — | — | — | — | 50.85 | 34.71 | 32.95 | 25.98 | 37.78 | 28.57 | — | — | — | — | — | |
| ParisKVBase Model=Qwen2.5-7B2026.02 | 33.07 | — | — | — | — | — | 52.54 | 34.71 | 34.09 | 26.77 | 37.21 | 19.67 | — | — | — | — | — | |
| S2OBackbone=Qwen3-8B2026.02 | 32.6 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| X-AttnBackbone=Qwen3-8B2026.02 | 32 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| PBSBackbone=Qwen3-8B2026.02 | 32 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| S2OBackbone=Llama-3.1-8B2026.02 | 31.41 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| DuoAttentionbackbone=LLaMA-3.1-8B-Instruct2026.05 | 31.2 | 31.2 | 31.2 | 36.1 | 27 | 31.5 | — | — | — | — | — | — | — | — | — | — | — | |
| PBSBackbone=Llama-3.1-8B2026.02 | 30.41 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| MAC-AttentionKV %=52026.03 | 30.4 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| KVSlimmer2026.03 | 30.22 | 32.81 | 28.62 | 36.11 | 25.12 | 30.56 | — | — | — | — | — | — | — | — | — | — | — | |
| FullBackbone=Llama-3.1-8B2026.02 | 30.22 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| FlexPrefillBackbone=Qwen3-8B2026.02 | 30.21 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| MultipoleKV %=102026.03 | 30.2 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| MAC-AttentionKV %=12026.03 | 30.2 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| MAC-AttentionKV %=102026.03 | 30.2 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Full Context2026.03 | 30.02 | 30.73 | 29.58 | 35 | 27.91 | 25.93 | — | — | — | — | — | — | — | — | — | — | — | |
| AsymKV2026.03 | 30.02 | 30.23 | 29.9 | 32.78 | 27.44 | 28.85 | — | — | — | — | — | — | — | — | — | — | — | |
| Llama-3.1-8B-Instructbackbone=LLaMA-3.1-8B-Instruct2026.05 | 30 | 30.7 | 29.6 | 35 | 26.5 | 28.7 | — | — | — | — | — | — | — | — | — | — | — | |
| X-AttnBackbone=Llama-3.1-8B2026.02 | 29.82 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Training–Inference Consistent Segmented Executionbackbone=LLaMA-3.1-8B-Instruct2026.05 | 29.8 | 27.6 | 31.2 | 33.3 | 28.4 | 26.9 | — | — | — | — | — | — | — | — | — | — | — | |
| MAC-AttentionKV %=202026.03 | 29.6 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| RocketKVKV %=12026.03 | 29.4 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| RocketKVKV %=202026.03 | 29.4 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| RocketKVKV %=52026.03 | 29.2 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| RocketKVKV %=102026.03 | 29.2 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Full Attn.KV %=12026.03 | 29 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Full Attn.KV %=52026.03 | 29 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Full Attn.KV %=102026.03 | 29 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Full Attn.KV %=202026.03 | 29 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| BAM SSMaxModel Size=1B2025.05 | 28.6 | — | — | — | — | — | — | — | — | — | — | — | 41.7 | 36.4 | 35 | 26.5 | 26.5 | |
| ParisKVBase Model=DS-R1-Llama3-8B2026.02 | 28.43 | — | — | — | — | — | 37.29 | 25.62 | 28.41 | 25.2 | 31.11 | 30.16 | — | — | — | — | — | |
| LongCache2026.03 | 28.43 | 28.13 | 28.62 | 32.78 | 25.58 | 26.85 | — | — | — | — | — | — | — | — | — | — | — | |
| H2O2026.03 | 28.23 | 28.12 | 28.29 | 31.67 | 26.98 | 25 | — | — | — | — | — | — | — | — | — | — | — | |
| CaM2026.03 | 28.23 | 28.64 | 27.97 | 31.67 | 26.98 | 25 | — | — | — | — | — | — | — | — | — | — | — | |
| QuestKV %=202026.03 | 28.2 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| MultipoleKV %=202026.03 | 28 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| QuestKV %=52026.03 | 27.8 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| MultipoleKV %=52026.03 | 27.8 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| QuestKV %=12026.03 | 27.6 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| QuestKV %=102026.03 | 27.6 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| MultipoleKV %=12026.03 | 27.6 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| MInferencebackbone=LLaMA-3.1-8B-Instruct2026.05 | 27.6 | 27.1 | 28 | 33.3 | 24.2 | 25 | — | — | — | — | — | — | — | — | — | — | — | |
| StreamingLLM2026.03 | 27.04 | 27.6 | 26.69 | 32.78 | 23.26 | 25 | — | — | — | — | — | — | — | — | — | — | — | |
| FlexPrefillBackbone=Llama-3.1-8B2026.02 | 26.44 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Qwen2-7BBase Model=Qwen2-7B2026.02 | 25.84 | — | — | — | — | — | 27.12 | 16.53 | 36.36 | 25.2 | 26.67 | 28.57 | — | — | — | — | — | |
| PQCacheBase Model=Qwen2.5-7B2026.02 | 25.5 | — | — | — | — | — | 23.7 | 31.6 | 28.2 | 24 | 25 | 20 | — | — | — | — | — | |
| ParisKVBase Model=Qwen2-7B2026.02 | 24.6 | — | — | — | — | — | 35.59 | 19.49 | 26.14 | 22.05 | 28.89 | 23.81 | — | — | — | — | — | |
| RoPE SSMaxModel Size=1B2025.05 | 24.2 | — | — | — | — | — | — | — | — | — | — | — | 25 | 30.3 | 35 | 25.3 | 18.8 | |
| StreamingLLMbackbone=LLaMA-3.1-8B-Instruct2026.05 | 23.5 | 22.9 | 23.8 | 30 | 18.6 | 22.2 | — | — | — | — | — | — | — | — | — | — | — | |
| PQCacheBase Model=DS-R1-Llama3-8B2026.02 | 19.9 | — | — | — | — | — | 18.6 | 21.5 | 21.6 | 22.2 | 15.6 | 14.3 | — | — | — | — | — | |
| PQCacheBase Model=Qwen2-7B2026.02 | 17.91 | — | — | — | — | — | 16.95 | 19 | 13.6 | 19 | 20 | 19.05 | — | — | — | — | — | |
| MagicPIGBase Model=Qwen2-7B2026.02 | 16.7 | — | — | — | — | — | 18.64 | 10.74 | 14.77 | 20.47 | 28.89 | 12.7 | — | — | — | — | — | |
| MagicPIGBase Model=DS-R1-Llama3-8B2026.02 | 13.92 | — | — | — | — | — | 15.25 | 11.57 | 11.36 | 14.17 | 17.78 | 17.46 | — | — | — | — | — | |
| DS-R1-Llama3-8BBase Model=DS-R1-Llama3-8B2026.02 | 13.12 | — | — | — | — | — | 18.64 | 15.7 | 12.5 | 8.66 | 11.11 | 14.29 | — | — | — | — | — | |
| MagicPIGBase Model=Qwen2.5-7B2026.02 | 10.34 | — | — | — | — | — | 8.47 | 15.7 | 7.95 | 7.87 | 13.33 | 7.94 | — | — | — | — | — |