Long-context language modeling and retrieval on RULER
96.4VT ScoreAPB
Evaluation results
| Method | Links | ||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| APBBackbone=Qwen-2.5-14B-instruct2025.02 | 96.4 | 100 | 100 | 99.8 | 93 | 87.2 | 85.8 | 66.55 | 93 | 94.94 | 76.33 | 60 | 55.6 | 85.28 | — | — | — | — | |
| FULLATTNBackbone=Qwen-2.5-14B-instruct2025.02 | 89.52 | 100 | 99.2 | 99.8 | 94.2 | 47.8 | 27.2 | 75.1 | 94.6 | 93.88 | 76.13 | 63.2 | 43.4 | 77.23 | — | — | — | — | |
| MINFERENCEBackbone=Qwen-2.5-14B-instruct2025.02 | 89.44 | 100 | 99.6 | 100 | 89.6 | 32.6 | 6.94 | 61.25 | 93.29 | 84.36 | 77.33 | 56.6 | 44.4 | 71.95 | — | — | — | — | |
| FULLATTNBackbone=Yi-34B-200K2025.02 | 85.56 | 100 | 100 | 99.6 | 95.2 | 76 | 55.4 | 92.1 | 97.05 | 51.84 | 84.27 | 65.2 | 50 | 80.94 | — | — | — | — | |
| STARATTNBackbone=Yi-34B-200K2025.02 | 83.96 | 100 | 100 | 99.6 | 93.2 | 78.2 | 48 | 77.05 | 88.35 | 80.58 | 78.33 | 61.2 | 48.8 | 79.79 | — | — | — | — | |
| STARATTNBackbone=Qwen-2.5-14B-instruct2025.02 | 82.44 | 100 | 99.2 | 97.4 | 80 | 38 | 20.2 | 48.75 | 77.1 | 93.86 | 77.53 | 52.8 | 38 | 69.64 | — | — | — | — | |
| MINFERENCEBackbone=Yi-34B-200K2025.02 | 73.68 | 100 | 100 | 100 | 96 | 87 | 59.6 | 90.15 | 96.05 | 79.2 | 84.87 | 67.4 | 49.4 | 83.33 | — | — | — | — | |
| MINFERENCEBackbone=Llama-3.1-8B-instruct2025.02 | 66.4 | 100 | 98.6 | 99 | 95.4 | 58.2 | 23.8 | 84.35 | 95.7 | 45.94 | 74.67 | 67.8 | 38.8 | 72.97 | — | — | — | — | |
| FULLATTNBackbone=Llama-3.1-8B-instruct2025.02 | 60.98 | 99.4 | 99.8 | 99.6 | 98.2 | 87.6 | 67 | 94.65 | 98 | 71.4 | 72.2 | 78.2 | 41.6 | 82.2 | — | — | — | — | |
| APBBackbone=Yi-34B-200K2025.02 | 60.48 | 100 | 100 | 100 | 92.8 | 88 | 92.6 | 86.05 | 96.45 | 88.46 | 84.53 | 63.8 | 52.2 | 85.03 | — | — | — | — | |
| STARATTNBackbone=Llama-3.1-8B-instruct2025.02 | 59.4 | 100 | 99.6 | 99.6 | 95.8 | 73.6 | 53 | 72.8 | 94.45 | 65.72 | 76.53 | 67.4 | 41 | 76.84 | — | — | — | — | |
| APBBackbone=Llama-3.1-8B-instruct2025.02 | 51.96 | 100 | 100 | 99.8 | 85.6 | 91 | 89 | 95.05 | 96.4 | 63.82 | 77.33 | 70 | 41.2 | 81.63 | — | — | — | — | |
| Full AttentionAttention mechanism=Full Attention, Budget=Full context, Backbone=Llama3-8B-Instruct-Gradient-1048k2026.02 | 19.4 | — | — | — | — | — | — | — | — | — | 40 | 75 | 56.4 | 63.3 | 100 | 95 | 90.3 | 70 | |
| LycheeDecodeAttention mechanism=Sparse Attention, Budget=4096 tokens, Backbone=Llama3-8B-Instruct-Gradient-1048k2026.02 | 19.4 | — | — | — | — | — | — | — | — | — | 40 | 75.4 | 56.2 | 62.8 | 100 | 94.4 | 90.6 | 65.9 |