Long-context language understanding on LongBench-E 2024 (test)
7.58Short Context QA ScoreTraining–Inference Consistent Segmented Execution
Evaluation results
| Method | Links | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| Training–Inference Consistent Segmented ExecutionBackbone=LLaMA2-7B-80K, Prefill Length=32K, Context Window=80K2026.05 | 7.58 | 8.56 | 8.7 | 63.76 | 0.42 | 56.02 | 24.17 | 3.49 | 19.06 | |
| MInferenceBackbone=LLaMA2-7B-80K, Prefill Length=32K, Context Window=80K2026.05 | 6.93 | 5.96 | 11.19 | 66.42 | 1.01 | 48.6 | 23.35 | 4.13 | 31.77 | |
| LLaMA2-7B-80K (Vanilla Self-Attention)Backbone=LLaMA2-7B-80K, Prefill Length=32K, Context Window=80K2026.05 | 6.89 | 5.93 | 11.35 | 66.51 | 0.77 | 48.83 | 23.38 | 4.13 | 34.67 | |
| DuoAttentionBackbone=LLaMA2-7B-80K, Prefill Length=32K, Context Window=80K2026.05 | 6.57 | 6.12 | 11.86 | 63.68 | 0.89 | 48.5 | 22.94 | 3.79 | 23.66 | |
| CCA-attentionBackbone=LLaMA2-7B-80K, Prefill Length=32K, Context Window=80K2026.05 | 6.22 | 7.86 | 6.11 | 58.2 | 1.73 | 51.75 | 21.98 | 3.88 | 43.64 | |
| StreamingLLMBackbone=LLaMA2-7B-80K, Prefill Length=32K, Context Window=80K2026.05 | 3.48 | 3.76 | 6.6 | 62.35 | 0.17 | 53.02 | 21.56 | 3.07 | 31.77 | |
| CCA-attentionBackbone=LLaMA2-7B-32K, Prefill Length=16K, Context Window=32K2026.05 | 3.36 | 5.64 | 4.6 | 55.41 | 2.12 | 55.56 | 21.12 | 1.79 | 28.08 | |
| MInferenceBackbone=LLaMA2-7B-32K, Prefill Length=16K, Context Window=32K2026.05 | 2.87 | 6.39 | 3.56 | 65.44 | 0 | 60.22 | 23.08 | 2.84 | 22.19 | |
| DuoAttentionBackbone=LLaMA2-7B-32K, Prefill Length=16K, Context Window=32K2026.05 | 2.85 | 6.3 | 3.45 | 64.82 | 0.67 | 59.91 | 23 | 1.53 | 18.15 | |
| LLaMA2-7B-32K (Vanilla Self-Attention)Backbone=LLaMA2-7B-32K, Prefill Length=16K, Context Window=32K2026.05 | 2.82 | 6.09 | 3.09 | 65.36 | 0.5 | 60.91 | 23.13 | 1.62 | 23.61 | |
| StreamingLLMBackbone=LLaMA2-7B-32K, Prefill Length=16K, Context Window=32K2026.05 | 1.93 | 5.22 | 3.95 | 61.93 | 0.25 | 58.12 | 21.9 | 1.59 | 22.19 | |
| Training–Inference Consistent Segmented ExecutionBackbone=LLaMA2-7B-32K, Prefill Length=16K, Context Window=32K2026.05 | 1.9 | 4.07 | 12.1 | 62.3 | 0.28 | 58.81 | 23.24 | 1.7 | 18.56 |