Long-context language understanding on RULER 16k context length
0FWE ScoreFull Attention
Evaluation results
| Method | Links | ||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Full AttentionAttention mechanism=Full Attention, Budget=Full context, Backbone=Llama3-8B-Instruct-Gradient-1048k2026.02 | 0 | 100 | 93 | 95.7 | 81 | 19.8 | 74.2 | 53.4 | 64.6 | — | — | — | |
| LycheeDecodeAttention mechanism=Sparse Attention, Budget=4096 tokens, Backbone=Llama3-8B-Instruct-Gradient-1048k2026.02 | 0 | 100 | 81.8 | 96.3 | 68.7 | 19.6 | 71 | 53.4 | 61.4 | — | — | — | |
| E2LLM-RContext Length=16K2024.09 | 2.33 | — | — | — | — | 5.4 | — | — | — | 6 | 6.4 | 40 | |
| LLoCOContext Length=16K2024.09 | 2.67 | — | — | — | — | 0 | — | — | — | 0.41 | 20.9 | 20 | |
| E2LLM-C+RContext Length=16K2024.09 | 3 | — | — | — | — | 10.2 | — | — | — | 57.78 | 0.7 | 1 | |
| LongLoRAContext Length=16K2024.09 | 4 | — | — | — | — | 2 | — | — | — | 36.78 | 5.8 | 52 | |
| YaRNContext Length=16K2024.09 | 5.33 | — | — | — | — | 1.4 | — | — | — | 38.88 | 3.9 | 29 | |
| E2LLM-CContext Length=16K2024.09 | 10.33 | — | — | — | — | 1.6 | — | — | — | 2.06 | 7.3 | 28.5 | |
| LongLLMLinguaContext Length=16K2024.09 | 27.33 | — | — | — | — | 0 | — | — | — | 32.75 | 2 | 15.5 | |
| CCA-attentionBackbone=LLaMA2-7B-32K2026.05 | 29.83 | — | — | — | — | — | — | — | — | — | 15.9 | — | |
| StreamingLLMBackbone=LLaMA2-7B-32K2026.05 | 43 | — | — | — | — | — | — | — | — | — | 19.35 | — | |
| Training–Inference Consistent Segmented ExecutionBackbone=LLaMA2-7B-32K2026.05 | 44.83 | — | — | — | — | — | — | — | — | — | 41.45 | — | |
| MInferenceBackbone=LLaMA2-7B-32K2026.05 | 46.17 | — | — | — | — | — | — | — | — | — | 20.15 | — | |
| LLaMA2-7B-32K (Vanilla Self-Attention)Backbone=LLaMA2-7B-32K, Attention=Vanilla Self-Attention2026.05 | 47.5 | — | — | — | — | — | — | — | — | — | 20.05 | — | |
| DuoAttentionBackbone=LLaMA2-7B-32K2026.05 | 48.33 | — | — | — | — | — | — | — | — | — | 18.15 | — | |
| SOCKETModel=Llama-3.1-8B-Instruct, Sparsity=10x, P=10-12, L=60, τ=0.3-0.72026.02 | 88.67 | 100 | 100 | 98.5 | 98.25 | 88.8 | 88 | 54 | — | — | — | — | |
| HashAttentionModel=Llama-3.1-8B-Instruct, Sparsity=10x2026.02 | 91.33 | 100 | 100 | 98 | 99 | 89 | 73 | 45.5 | — | — | — | — | |
| dense attentionModel=Llama-3.1-8B-Instruct, Sparsity=None (Dense)2026.02 | 93.17 | 100 | 100 | 98.5 | 97 | 97.4 | 80.5 | 51.5 | — | — | — | — | |
| vAttention (oracle-top-k)Model=Llama-3.1-8B-Instruct, Sparsity=10x2026.02 | 93.17 | 100 | 100 | 98 | 97 | 97.5 | 79.5 | 51.5 | — | — | — | — | |
| oracle-top-kModel=Llama-3.1-8B-Instruct, Sparsity=10x2026.02 | 93.17 | 100 | 100 | 97.5 | 98.5 | 97.6 | 73.5 | 48 | — | — | — | — | |
| vAttention (HashAttention)Model=Llama-3.1-8B-Instruct, Sparsity=10x2026.02 | 93.83 | 100 | 100 | 94 | 98 | 96.2 | 76 | 48 | — | — | — | — | |
| HATA AlgoModel=Qwen2.5-14B2025.11 | — | — | — | — | — | — | — | — | 94.06 | — | — | — | |
| HATA AlgoModel=Llama3-8B2025.11 | — | — | — | — | — | — | — | — | 86.56 | — | — | — | |
| InfiniGenModel=Qwen2.5-14B2025.11 | — | — | — | — | — | — | — | — | 92.78 | — | — | — | |
| InfiniGenModel=Llama3-8B2025.11 | — | — | — | — | — | — | — | — | 79.7 | — | — | — | |
| LiteCache + HATAModel=Qwen2.5-14B2025.11 | — | — | — | — | — | — | — | — | 94.36 | — | — | — | |
| LiteCache + HATAModel=Llama3-8B2025.11 | — | — | — | — | — | — | — | — | 85.65 | — | — | — | |
| Llama3-8BModel=Llama3-8B2025.11 | — | — | — | — | — | — | — | — | 86.07 | — | — | — | |
| Qwen 1.5 (1.8B)Configuration=Origin2026.01 | — | — | — | — | — | — | — | — | 40.2 | — | — | — | |
| Qwen 1.5 (1.8B)Configuration=Recap2026.01 | — | — | — | — | — | — | — | — | 50.6 | — | — | — | |
| Qwen2.5-14BModel=Qwen2.5-14B2025.11 | — | — | — | — | — | — | — | — | 94.35 | — | — | — | |
| RetroInferModel=Qwen2.5-14B2025.11 | — | — | — | — | — | — | — | — | 94.73 | — | — | — | |
| RetroInferModel=Llama3-8B2025.11 | — | — | — | — | — | — | — | — | 86.36 | — | — | — | |
| RWKV7 (1.5B)Configuration=Origin2026.01 | — | — | — | — | — | — | — | — | 20.6 | — | — | — | |
| RWKV7 (1.5B)Configuration=Recap2026.01 | — | — | — | — | — | — | — | — | 23.4 | — | — | — |