Long-context language understanding on RULER 32k context length
0FWEFull Attention
Evaluation results
| Method | Links | ||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Full AttentionAttention mechanism=Full Attention, Budget=Full context, Backbone=Llama3-8B-Instruct-Gradient-1048k2026.02 | 0 | 99.2 | 97.4 | 96.5 | 81.9 | 19.8 | 70.6 | 51.6 | 65.9 | — | — | — | — | — | |
| LycheeDecodeAttention mechanism=Sparse Attention, Budget=4096 tokens, Backbone=Llama3-8B-Instruct-Gradient-1048k2026.02 | 0 | 97.8 | 82 | 94.9 | 65.1 | 19.8 | 66.2 | 49.6 | 59.4 | — | — | — | — | — | |
| LongLoRAContext Length=32K2024.09 | 1.67 | — | — | — | — | 0.4 | — | — | — | — | — | 34.75 | 1.8 | 33.5 | |
| E2LLM-RContext Length=32K2024.09 | 1.67 | — | — | — | — | 5.6 | — | — | — | — | — | 8.13 | 5.4 | 4 | |
| YaRNContext Length=32K2024.09 | 2 | — | — | — | — | 1.2 | — | — | — | — | — | 32.94 | 2.8 | 28.5 | |
| E2LLM-C+RContext Length=32K2024.09 | 5 | — | — | — | — | 8.6 | — | — | — | — | — | 52.66 | 6.7 | 40.5 | |
| LongLLMLinguaContext Length=32K2024.09 | 11.33 | — | — | — | — | 0.2 | — | — | — | — | — | 26.06 | 0.3 | 18.5 | |
| E2LLM-CContext Length=32K2024.09 | 12.67 | — | — | — | — | 0.8 | — | — | — | — | — | 2.36 | 0.5 | 28 | |
| CCA-attentionBackbone=LLaMA2-7B-32K2026.05 | 12.83 | — | — | — | — | — | — | — | — | — | — | — | 6.35 | — | |
| LLoCOContext Length=32K2024.09 | 24 | — | — | — | — | 0 | — | — | — | — | — | 0.25 | 0.1 | 4.5 | |
| DuoAttentionBackbone=LLaMA2-7B-32K2026.05 | 33.83 | — | — | — | — | — | — | — | — | — | — | — | 12.2 | — | |
| LLaMA2-7B-32K (Vanilla Self-Attention)Backbone=LLaMA2-7B-32K, Attention=Vanilla Self-Attention2026.05 | 34 | — | — | — | — | — | — | — | — | — | — | — | 11.8 | — | |
| StreamingLLMBackbone=LLaMA2-7B-32K2026.05 | 35.83 | — | — | — | — | — | — | — | — | — | — | — | 9.5 | — | |
| MInferenceBackbone=LLaMA2-7B-32K2026.05 | 36 | — | — | — | — | — | — | — | — | — | — | — | 11.85 | — | |
| Training–Inference Consistent Segmented ExecutionBackbone=LLaMA2-7B-32K2026.05 | 38.17 | — | — | — | — | — | — | — | — | — | — | — | 19.35 | — | |
| MagicPigSparsity=50x, Additional Memory (bits/token)=1024, Backbone=Llama-3.1-8B-Instruct2026.02 | 61.7 | — | — | — | — | 0 | 11 | 24 | 16.2 | 1 | 0 | — | — | — | |
| SOCKETSparsity=50x, Additional Memory (bits/token)=600, Backbone=Llama-3.1-8B-Instruct2026.02 | 64 | — | — | — | — | 83.6 | 77 | 50 | 71.9 | 83 | 74 | — | — | — | |
| QuestSparsity=50x, Additional Memory (bits/token)=512, Backbone=Llama-3.1-8B-Instruct2026.02 | 80 | — | — | — | — | 76 | 71 | 54 | 64.1 | 74 | 30 | — | — | — | |
| MagicPigSparsity=10x, Additional Memory (bits/token)=1024, Backbone=Llama-3.1-8B-Instruct2026.02 | 83 | — | — | — | — | 32.2 | 35 | 29 | 30.2 | 2 | 0 | — | — | — | |
| MagicPigSparsity=20x, Additional Memory (bits/token)=1024, Backbone=Llama-3.1-8B-Instruct2026.02 | 83 | — | — | — | — | 0 | 35 | 29 | 24.8 | 2 | 0 | — | — | — | |
| PQcacheSparsity=50x, Additional Memory (bits/token)=256, Backbone=Llama-3.1-8B-Instruct2026.02 | 83.3 | — | — | — | — | 86 | 73 | 49 | 69.5 | 64 | 62 | — | — | — | |
| HashAttnSparsity=50x, Additional Memory (bits/token)=128, Backbone=Llama-3.1-8B-Instruct2026.02 | 83.7 | — | — | — | — | 72.2 | 63 | 46 | 49.4 | 32 | 0 | — | — | — | |
| SOCKETSparsity=20x, Additional Memory (bits/token)=600, Backbone=Llama-3.1-8B-Instruct2026.02 | 86 | — | — | — | — | 91.4 | 82 | 52 | 82.7 | 93 | 92 | — | — | — | |
| PQcacheSparsity=20x, Additional Memory (bits/token)=256, Backbone=Llama-3.1-8B-Instruct2026.02 | 87.7 | — | — | — | — | 94.2 | 82 | 50 | 83.1 | 93 | 92 | — | — | — | |
| QuestSparsity=20x, Additional Memory (bits/token)=512, Backbone=Llama-3.1-8B-Instruct2026.02 | 87.7 | — | — | — | — | 91.4 | 84 | 54 | 83.8 | 90 | 96 | — | — | — | |
| QuestSparsity=10x, Additional Memory (bits/token)=512, Backbone=Llama-3.1-8B-Instruct2026.02 | 88.3 | — | — | — | — | 95 | 86 | 53 | 86.8 | 100 | 99 | — | — | — | |
| DSSparsity=50x, Additional Memory (bits/token)=512, Backbone=Llama-3.1-8B-Instruct2026.02 | 88.3 | — | — | — | — | 66.4 | 48 | 44 | 44.6 | 20 | 1 | — | — | — | |
| SOCKETSparsity=10x, Additional Memory (bits/token)=600, Backbone=Llama-3.1-8B-Instruct2026.02 | 88.7 | — | — | — | — | 94.2 | 82 | 53 | 85.5 | 95 | 100 | — | — | — | |
| QuestSparsity=5x, Additional Memory (bits/token)=512, Backbone=Llama-3.1-8B-Instruct2026.02 | 89.7 | — | — | — | — | 97.6 | 85 | 54 | 87.5 | 100 | 99 | — | — | — | |
| HashAttnSparsity=20x, Additional Memory (bits/token)=128, Backbone=Llama-3.1-8B-Instruct2026.02 | 89.7 | — | — | — | — | 81 | 75 | 51 | 69.1 | 73 | 45 | — | — | — | |
| MagicPigSparsity=5x, Additional Memory (bits/token)=1024, Backbone=Llama-3.1-8B-Instruct2026.02 | 91.7 | — | — | — | — | 82.8 | 38 | 42 | 44.1 | 10 | 0 | — | — | — | |
| SOCKETSparsity=5x, Additional Memory (bits/token)=600, Backbone=Llama-3.1-8B-Instruct2026.02 | 91.7 | — | — | — | — | 95.2 | 84 | 53 | 86.8 | 97 | 100 | — | — | — | |
| PQcacheSparsity=10x, Additional Memory (bits/token)=256, Backbone=Llama-3.1-8B-Instruct2026.02 | 92.3 | — | — | — | — | 96.4 | 81 | 51 | 86.2 | 98 | 99 | — | — | — | |
| DSSparsity=10x, Additional Memory (bits/token)=512, Backbone=Llama-3.1-8B-Instruct2026.02 | 92.3 | — | — | — | — | 95.2 | 74 | 51 | 83.4 | 91 | 97 | — | — | — | |
| HashAttnSparsity=10x, Additional Memory (bits/token)=128, Backbone=Llama-3.1-8B-Instruct2026.02 | 92.3 | — | — | — | — | 88.6 | 79 | 50 | 80.4 | 87 | 86 | — | — | — | |
| DSSparsity=20x, Additional Memory (bits/token)=512, Backbone=Llama-3.1-8B-Instruct2026.02 | 92.3 | — | — | — | — | 90.2 | 68 | 51 | 72 | 49 | 82 | — | — | — | |
| PQcacheSparsity=5x, Additional Memory (bits/token)=256, Backbone=Llama-3.1-8B-Instruct2026.02 | 92.7 | — | — | — | — | 98.2 | 81 | 51 | 86.9 | 100 | 99 | — | — | — | |
| DSSparsity=5x, Additional Memory (bits/token)=512, Backbone=Llama-3.1-8B-Instruct2026.02 | 93 | — | — | — | — | 97.8 | 81 | 51 | 85.3 | 91 | 98 | — | — | — | |
| HashAttnSparsity=5x, Additional Memory (bits/token)=128, Backbone=Llama-3.1-8B-Instruct2026.02 | 93.7 | — | — | — | — | 94.2 | 83 | 49 | 85.6 | 97 | 97 | — | — | — | |
| Qwen 1.5 (1.8B)Configuration=Origin2026.01 | — | — | — | — | — | — | — | — | 9.3 | — | — | — | — | — | |
| Qwen 1.5 (1.8B)Configuration=Recap2026.01 | — | — | — | — | — | — | — | — | 24.1 | — | — | — | — | — | |
| RWKV7 (1.5B)Configuration=Origin2026.01 | — | — | — | — | — | — | — | — | 9.2 | — | — | — | — | — | |
| RWKV7 (1.5B)Configuration=Recap2026.01 | — | — | — | — | — | — | — | — | 11.7 | — | — | — | — | — |