Long-context Understanding on LongBench (Full Suite Evaluation)
63.4Average ScoreLongCrafter
Evaluation results
| Method | Links | |||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| LongCrafterBackbone=LLaMA-3.1-8B, SFT Method=LongCrafter2026.07 | 63.4 | — | 53 | — | 72.5 | 70.5 | 57.5 | — | — | — | — | — | — | — | — | — | — | |
| LongCrafterBackbone=Qwen2.5-7B, SFT Method=LongCrafter2026.07 | 62.8 | — | 52.5 | — | 75.5 | 74 | 49 | — | — | — | — | — | — | — | — | — | — | |
| LongFaithBackbone=LLaMA-3.1-8B, SFT Method=LongFaith2026.07 | 60.5 | — | 45.5 | — | 69 | 71 | 56.5 | — | — | — | — | — | — | — | — | — | — | |
| LongFaithBackbone=Qwen2.5-7B, SFT Method=LongFaith2026.07 | 59.4 | — | 47 | — | 73.5 | 71 | 46 | — | — | — | — | — | — | — | — | — | — | |
| LongMagpieBackbone=LLaMA-3.1-8B, SFT Method=LongMagpie2026.07 | 56.9 | — | 51.5 | — | 67.5 | 62.5 | 46 | — | — | — | — | — | — | — | — | — | — | |
| Qwen2.5-7B-InstructBackbone=Qwen2.5-7B, SFT Method=Instruct (official)2026.07 | 56.8 | — | 57.5 | — | 74 | 60 | 35.5 | — | — | — | — | — | — | — | — | — | — | |
| LLaMA-3.1-8B-InstructBackbone=LLaMA-3.1-8B, SFT Method=Instruct (official)2026.07 | 53.25 | — | 53 | — | 69 | 58 | 33 | — | — | — | — | — | — | — | — | — | — | |
| LongMagpieBackbone=Qwen2.5-7B, SFT Method=LongMagpie2026.07 | 52.9 | — | 50.5 | — | 67 | 55 | 39 | — | — | — | — | — | — | — | — | — | — | |
| LongRewardBackbone=Qwen2.5-7B, SFT Method=LongReward2026.07 | 52.5 | — | 50.5 | — | 67.5 | 54 | 38 | — | — | — | — | — | — | — | — | — | — | |
| LongRewardBackbone=LLaMA-3.1-8B, SFT Method=LongReward2026.07 | 49.4 | — | 50 | — | 63.5 | 45.5 | 38.5 | — | — | — | — | — | — | — | — | — | — | |
| LongAlignBackbone=Qwen2.5-7B, SFT Method=LongAlign2026.07 | 48.9 | — | 51 | — | 63.5 | 51.5 | 29.5 | — | — | — | — | — | — | — | — | — | — | |
| FullBackbone=Mistral-7B-Instruct-v0.3, Cache Budget=Full Cache2026.05 | 48.11 | 29.07 | 41.54 | 52.88 | 49.37 | 39.01 | 28.58 | 35.07 | 25.71 | 27.73 | 76 | 88.59 | 47.51 | 6 | 98.5 | 61.48 | 62.68 | |
| HeadKV w/ CriticalKVBackbone=Llama-3.1-8B, Cache Budget=40%2025.02 | 48 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| AdaKV w/ CriticalKVBackbone=Llama-3.1-8B, Cache Budget=40%2025.02 | 47.79 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| SnapKV w/ CriticalKVBackbone=Llama-3.1-8B, Cache Budget=40%2025.02 | 47.29 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| LongAlignBackbone=LLaMA-3.1-8B, SFT Method=LongAlign2026.07 | 46.6 | — | 51 | — | 60 | 45 | 30.5 | — | — | — | — | — | — | — | — | — | — | |
| ReST-KVBackbone=Mistral-7B-Instruct-v0.3, Cache Budget=512L2026.05 | 46.47 | 28.6 | 35.86 | 53.37 | 49.13 | 38.7 | 27.94 | 26.05 | 24.37 | 25.09 | 73.5 | 89.66 | 46.27 | 5.5 | 98.5 | 60.13 | 60.84 | |
| SnapKV w/ OBCache (Joint)Backbone=Llama-3.1-8B, Cache Budget=40%2025.02 | 45.92 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| SnapKVBackbone=Mistral-7B-Instruct-v0.3, Cache Budget=512L2026.05 | 45.66 | 26.84 | 35.51 | 53.12 | 49.56 | 37.72 | 26.54 | 25.06 | 24.03 | 24.76 | 67.5 | 89.36 | 44.82 | 5.5 | 98.5 | 60.44 | 61.22 | |
| SnapKVBackbone=Llama-3.1-8B, Cache Budget=40%2025.02 | 45.52 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| TOVABackbone=Mistral-7B-Instruct-v0.3, Cache Budget=512L2026.05 | 42.85 | 25.23 | 32.52 | 46.24 | 45.23 | 36.23 | 20.32 | 24.53 | 22.53 | 23.64 | 66.5 | 87.24 | 44.21 | 6 | 85.62 | 59.35 | 60.24 | |
| H2OBackbone=Mistral-7B-Instruct-v0.3, Cache Budget=512L2026.05 | 41.42 | 25.23 | 30.41 | 40.32 | 42.52 | 35.23 | 18.23 | 24.23 | 21.24 | 23.21 | 66.5 | 86.71 | 43.15 | 5 | 82.52 | 60.13 | 58.15 | |
| ReST-KVBackbone=Mistral-7B-Instruct-v0.3, Cache Budget=64L2026.05 | 40.05 | 25.65 | 26.58 | 42.71 | 46.11 | 36.43 | 24.34 | 19.8 | 21.65 | 18.9 | 51.5 | 87.88 | 41.54 | 4 | 90.5 | 52.39 | 50.75 | |
| FullBackbone=Llama-3.1-8B-Instruct, Cache Budget=Full Cache2026.05 | 39.96 | 32.02 | 13.12 | 27.52 | 16.6 | 16.41 | 11.41 | 34.59 | 23.41 | 26.89 | 73 | 91.65 | 43.8 | 7.18 | 97.73 | 65.12 | 58.89 | |
| StreamingLLMBackbone=Mistral-7B-Instruct-v0.3, Cache Budget=512L2026.05 | 39.68 | 24.19 | 25.97 | 30.14 | 40.75 | 31.9 | 17.35 | 22.18 | 20.3 | 23.22 | 65.5 | 86.95 | 43.75 | 6 | 81 | 59.35 | 56.36 | |
| ReST-KVBackbone=Llama-3.1-8B-Instruct, Cache Budget=512L2026.05 | 38.33 | 32.01 | 10.73 | 25.23 | 15.91 | 15.85 | 10.25 | 26.47 | 23.23 | 24.79 | 69 | 91.62 | 42.59 | 8.4 | 97.66 | 63.48 | 56.03 | |
| SnapKVBackbone=Mistral-7B-Instruct-v0.3, Cache Budget=64L2026.05 | 37.53 | 19.39 | 23.62 | 38.66 | 43.26 | 34.72 | 21.33 | 17.59 | 20.93 | 17.06 | 38.5 | 86.96 | 39.61 | 7 | 90.5 | 51.63 | 49.73 | |
| SnapKVBackbone=Llama-3.1-8B-Instruct, Cache Budget=512L2026.05 | 37.47 | 28.02 | 9.83 | 24.84 | 13.77 | 15.4 | 10.21 | 25.13 | 22.73 | 24.25 | 65 | 92.34 | 41.69 | 8.42 | 96.31 | 64.3 | 57.28 | |
| TOVABackbone=Mistral-7B-Instruct-v0.3, Cache Budget=64L2026.05 | 36.5 | 22.51 | 22.24 | 37.23 | 41.12 | 34.1 | 19.52 | 17.21 | 19.23 | 16.27 | 38.5 | 85.12 | 38.51 | 6.5 | 86.5 | 51.04 | 48.42 | |
| TOVABackbone=Llama-3.1-8B-Instruct, Cache Budget=512L2026.05 | 35.59 | 27.34 | 8.34 | 22.45 | 12.25 | 14.51 | 8.42 | 24.23 | 22.13 | 22.25 | 58.5 | 89.31 | 40.51 | 8.24 | 93.14 | 62.23 | 55.61 | |
| H2OBackbone=Mistral-7B-Instruct-v0.3, Cache Budget=64L2026.05 | 34.93 | 20.51 | 21.52 | 25.12 | 40.12 | 33.12 | 18.34 | 16.23 | 19.12 | 16.24 | 38.5 | 83.12 | 37.23 | 6 | 85.5 | 50.12 | 48.12 | |
| H2OBackbone=Llama-3.1-8B-Instruct, Cache Budget=512L2026.05 | 34.53 | 26.23 | 7.34 | 20.51 | 11.52 | 13.52 | 7.34 | 23.23 | 21.24 | 23.14 | 58.5 | 86.12 | 40.15 | 7.25 | 91.02 | 61.23 | 54.12 | |
| StreamingLLMBackbone=Mistral-7B-Instruct-v0.3, Cache Budget=64L2026.05 | 34.24 | 20.37 | 20.56 | 24.62 | 38.87 | 32.47 | 17.68 | 15.48 | 19.84 | 15.81 | 39.5 | 82.77 | 36.72 | 5.5 | 80 | 49.77 | 47.9 | |
| StreamingLLMBackbone=Llama-3.1-8B-Instruct, Cache Budget=512L2026.05 | 33.84 | 19.15 | 6.47 | 15.02 | 10.94 | 12.58 | 6.23 | 23.66 | 20.05 | 23.31 | 57.5 | 87.7 | 41.86 | 10.25 | 90.74 | 62.39 | 53.61 | |
| ReST-KVBackbone=Llama-3.1-8B-Instruct, Cache Budget=64L2026.05 | 33.54 | 22.43 | 7.19 | 19.25 | 14.11 | 15.04 | 7.97 | 20.56 | 21.1 | 19.15 | 53.5 | 88.23 | 40.21 | 8.46 | 93.9 | 56.74 | 48.77 | |
| ReST-KVBackbone=Llama2-7B-Chat, Cache Budget=512L2026.05 | 32.38 | 17.15 | 19.88 | 32.71 | 31.94 | 25.62 | 9.97 | 20.52 | 20.68 | 23.59 | 63.5 | 83.3 | 39.29 | 6 | 11.5 | 58.65 | 53.81 | |
| SnapKVBackbone=Llama2-7B-Chat, Cache Budget=512L2026.05 | 32.18 | 16.22 | 19.57 | 32.32 | 31.87 | 24.97 | 9.66 | 20.19 | 20.77 | 23.85 | 62 | 82.24 | 39.18 | 6 | 10.5 | 59.49 | 56.06 | |
| SnapKVBackbone=Llama-3.1-8B-Instruct, Cache Budget=64L2026.05 | 31.48 | 19.9 | 5.78 | 18.38 | 13.51 | 14.42 | 8.52 | 17.35 | 20.44 | 17.33 | 41 | 85.37 | 37.63 | 8.93 | 91.08 | 55.09 | 48.88 | |
| TOVABackbone=Llama-3.1-8B-Instruct, Cache Budget=64L2026.05 | 30.63 | 18.52 | 6.12 | 17.32 | 12.15 | 12.51 | 7.35 | 16.24 | 20.41 | 16.34 | 38.41 | 82.61 | 36.16 | 8.14 | 95.23 | 55.21 | 47.35 | |
| TOVABackbone=Llama2-7B-Chat, Cache Budget=512L2026.05 | 30.24 | 13.53 | 15.46 | 26.44 | 26.12 | 31.02 | 7.12 | 18.25 | 18.64 | 22.34 | 62.5 | 83.1 | 40.61 | 3 | 8 | 56.14 | 51.53 | |
| StreamingLLMBackbone=Llama-3.1-8B-Instruct, Cache Budget=64L2026.05 | 29.61 | 7.65 | 5.08 | 14.14 | 10.93 | 12.64 | 6.86 | 16.57 | 18.93 | 16.3 | 38.5 | 83.13 | 34.65 | 9.78 | 96.28 | 54.16 | 48.21 | |
| H2OBackbone=Llama-3.1-8B-Instruct, Cache Budget=64L2026.05 | 29.52 | 12.23 | 5.12 | 15.12 | 11.51 | 10.14 | 6.23 | 17.23 | 19.51 | 16.79 | 39.15 | 81.51 | 36.12 | 8.12 | 95.12 | 51.25 | 47.12 | |
| StreamingLLMBackbone=Llama2-7B-Chat, Cache Budget=512L2026.05 | 28.86 | 15.3 | 15.53 | 20.16 | 26.59 | 25.05 | 5.65 | 18.3 | 19.28 | 21.84 | 54.5 | 82.23 | 38.07 | 5.5 | 5 | 56.8 | 51.95 | |
| ReST-KVBackbone=Llama2-7B-Chat, Cache Budget=64L2026.05 | 25.54 | 12.72 | 17.17 | 24.09 | 24.71 | 23.8 | 5.55 | 15.18 | 19.71 | 17.45 | 43.5 | 76.17 | 33.42 | 5.5 | 4 | 45 | 40.61 | |
| SnapKVBackbone=Llama2-7B-Chat, Cache Budget=64L2026.05 | 22.96 | 10.83 | 16.38 | 17.53 | 22.81 | 23.24 | 5.06 | 13.12 | 18.38 | 14.17 | 34.5 | 69.45 | 33.43 | 5.5 | 7 | 39.99 | 36.04 | |
| TOVABackbone=Llama2-7B-Chat, Cache Budget=64L2026.05 | 20.2 | 8.26 | 14.34 | 12.64 | 13.52 | 13.25 | 3.53 | 11.64 | 16.67 | 13.35 | 36 | 72.64 | 32.72 | 2 | 4 | 36.15 | 32.53 | |
| LaCacheBackbone=Llama2-7B-Chat, Cache Budget=64L2026.05 | 13.51 | 8.61 | 16.51 | 7.42 | 15.14 | 17.77 | 4.36 | 13.09 | 17.46 | 13.83 | 18.25 | 18.12 | 12.93 | 5.5 | 6 | 24 | 17.24 | |
| H2OBackbone=Llama2-7B-Chat, Cache Budget=512L2026.05 | 12.85 | 9.68 | 8.67 | 6.86 | 10.85 | 8.71 | 1.31 | 20.04 | 18.72 | 24.91 | 18 | 17.09 | 18.99 | 3.75 | 2.3 | 20.87 | 14.87 | |
| H2OBackbone=Llama2-7B-Chat, Cache Budget=64L2026.05 | 12.42 | 4.46 | 12.14 | 8.85 | 12.11 | 13.34 | 2.36 | 13.06 | 16.63 | 16.89 | 19.5 | 20.69 | 10.45 | 2.7 | 3 | 26.5 | 16.06 | |
| StreamingLLMBackbone=Llama2-7B-Chat, Cache Budget=64L2026.05 | 11.83 | 5.61 | 15.51 | 6.42 | 14.14 | 16.77 | 1.36 | 12.09 | 16.46 | 12.83 | 17.25 | 15.12 | 10.93 | 4.5 | 3 | 22 | 15.24 |