Long-context Understanding on RULER 8K context
54.15CWETraining–Inference Consistent Segmented Execution
Evaluation results
| Method | Links | |||||
|---|---|---|---|---|---|---|
| Training–Inference Consistent Segmented ExecutionBackbone=LLaMA2-7B-32K2026.05 | 54.15 | — | — | 38.67 | — | |
| MInferenceBackbone=LLaMA2-7B-32K2026.05 | 39.4 | — | — | 36.17 | — | |
| LLaMA2-7B-32K (Vanilla Self-Attention)Backbone=LLaMA2-7B-32K, Attention=Vanilla Self-Attention2026.05 | 39.25 | — | — | 35.83 | — | |
| DuoAttentionBackbone=LLaMA2-7B-32K2026.05 | 38.8 | — | — | 42.67 | — | |
| CCA-attentionBackbone=LLaMA2-7B-32K2026.05 | 36.25 | — | — | 39.5 | — | |
| LLoCOContext Length=8K2024.09 | 24.1 | 0.19 | 0 | 17 | 28.5 | |
| E2LLM-C+RContext Length=8K2024.09 | 21.1 | 60.66 | 8.6 | 12 | 5.5 | |
| StreamingLLMBackbone=LLaMA2-7B-32K2026.05 | 20.45 | — | — | 38.83 | — | |
| LongLoRAContext Length=8K2024.09 | 13.4 | 38.19 | 2.2 | 10.33 | 44 | |
| YaRNContext Length=8K2024.09 | 10.3 | 42.21 | 1.8 | 11.67 | 34.5 | |
| E2LLM-CContext Length=8K2024.09 | 6.3 | 3.5 | 0.6 | 11.67 | 31 | |
| LongLLMLinguaContext Length=8K2024.09 | 5.7 | 30.44 | 4.2 | 24.33 | 16 | |
| E2LLM-RContext Length=8K2024.09 | 1.5 | 9.5 | 0.6 | 9.33 | 33.5 |