Long-Context Language Modeling on RULER (4K to 128K Context Sweep)
96.29Accuracy (8K Context)Qwen2.5-14B-Instruct-1M
Evaluation results
| Method | Links | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| Qwen2.5-14B-Instruct-1MModel Scale=14B, Training Strategy=1M-Instruct2026.04 | 96.29 | 96.81 | 95.46 | 94.08 | 93.01 | 89.42 | 94.18 | — | — | — | |
| OPSDLModel Scale=32B, Training Strategy=OPSDL2026.04 | 96.29 | 96.56 | 96.17 | 95.81 | 91.04 | 84.31 | 93.36 | — | — | — | |
| Qwen2.5-32B-InstructModel Scale=32B, Training Strategy=Base Instruct2026.04 | 96.23 | 96.53 | 95.98 | 94.09 | 81.62 | 54.02 | 86.41 | — | — | — | |
| Long-SFTModel Scale=32B, Training Strategy=Long-SFT2026.04 | 95.62 | 96.05 | 95.21 | 94.1 | 89.72 | 80.7 | 91.9 | — | — | — | |
| Qwen2.5-14B-InstructModel Scale=14B, Training Strategy=Base Instruct2026.04 | 95.26 | 96.56 | 93.78 | 92.2 | 77.13 | 46.69 | 83.61 | — | — | — | |
| Long-SFTModel Scale=14B, Training Strategy=Long-SFT2026.04 | 94.98 | 95.96 | 92.92 | 90.47 | 84.62 | 73.69 | 88.77 | — | — | — | |
| OPSDLModel Scale=14B, Training Strategy=OPSDL2026.04 | 94.81 | 96.15 | 93.48 | 91.93 | 88.08 | 80.94 | 90.9 | — | — | — | |
| FullBackbone=Llama-3.1-8B2025.10 | 94.24 | 96.14 | 92.19 | 86.06 | 84.6 | 75.3 | 88.09 | — | — | — | |
| DENSEBackbone=LLAMA-3.1-8B-INSTRUCT2026.03 | 94.06 | 95.52 | 93.43 | 88.19 | 84.86 | 77.11 | 88.86 | 100 | — | — | |
| STEMBackbone=LLAMA-3.1-8B-INSTRUCT2026.03 | 93.96 | 95.17 | 93.73 | 89.92 | 82.67 | 75.37 | 88.47 | 25 | — | — | |
| XAttentionBackbone=Llama-3.1-8B2025.10 | 93.95 | 95.63 | 91.63 | 86.32 | 80.54 | 70.68 | 86.46 | — | — | — | |
| FLEXBackbone=LLAMA-3.1-8B-INSTRUCT2026.03 | 93.91 | 94.98 | 92.68 | 89.51 | 82.66 | 75.4 | 88.19 | 27 | — | — | |
| PBS-AttnBackbone=Llama-3.1-8B2025.10 | 93.85 | 95.83 | 91.46 | 85.18 | 82.51 | 66.98 | 85.97 | — | — | — | |
| PBS-Attn+Backbone=Llama-3.1-8B2025.10 | 93.85 | 95.72 | 91.23 | 87.05 | 81.27 | 72.09 | 86.87 | — | — | — | |
| XATTNBackbone=LLAMA-3.1-8B-INSTRUCT2026.03 | 93.83 | 95.46 | 93.65 | 89.82 | 81.6 | 74.35 | 88.12 | 26 | — | — | |
| MinferenceBackbone=Llama-3.1-8B2025.10 | 93.67 | 95.98 | 91.95 | 85.55 | 83.48 | 70.47 | 86.85 | — | — | — | |
| MINFBackbone=LLAMA-3.1-8B-INSTRUCT2026.03 | 93.56 | 95.51 | 92.7 | 91.68 | 80.06 | 76.67 | 88.36 | 55 | — | — | |
| LongPOModel Scale=7B, Training Strategy=LongPO2026.04 | 93.33 | 95.04 | 89.81 | 86.4 | 74.18 | 61.28 | 83.34 | — | — | — | |
| Qwen2.5-7B-Instruct-1MModel Scale=7B, Training Strategy=1M-Instruct2026.04 | 93.14 | 94.28 | 93.39 | 90.76 | 88.15 | 81.84 | 90.26 | — | — | — | |
| DENSEBackbone=QWEN3-8B2026.03 | 93.08 | 95.44 | 92.71 | 92.12 | 79.57 | 73.03 | 87.66 | 100 | — | — | |
| FlexPrefillBackbone=Llama-3.1-8B2025.10 | 92.99 | 92.87 | 91.35 | 84.91 | 82.62 | 71.07 | 85.97 | — | — | — | |
| Qwen2.5-7B-InstructModel Scale=7B, Training Strategy=Base Instruct2026.04 | 92.87 | 94.93 | 92.03 | 89.2 | 68.77 | 25.14 | 77.16 | — | — | — | |
| MeanPoolingBackbone=Llama-3.1-8B2025.10 | 92.72 | 94.15 | 89.94 | 83.95 | 76.46 | 59.32 | 82.76 | — | — | — | |
| STEMBackbone=QWEN3-8B2026.03 | 92.68 | 95.18 | 92.11 | 91.85 | 78.78 | 72.31 | 87.15 | 25 | — | — | |
| MINFBackbone=QWEN3-8B2026.03 | 92.56 | 95.01 | 91.7 | 91.68 | — | 71.87 | 86.73 | 76 | — | — | |
| XATTNBackbone=QWEN3-8B2026.03 | 92.56 | 95.03 | 91.76 | 91.72 | 78.6 | 71.8 | 86.91 | 35 | — | — | |
| FullBackbone=Qwen-2.5-7B-1M2025.10 | 92.45 | 95.34 | 93.49 | 89.06 | 84.73 | 74.23 | 88.22 | — | — | — | |
| OPSDLModel Scale=7B, Training Strategy=OPSDL2026.04 | 92.3 | 94.3 | 90.04 | 87.07 | 80.39 | 73.84 | 86.32 | — | — | — | |
| PBS-Attn+Backbone=Qwen-2.5-7B-1M2025.10 | 92.24 | 94.06 | 92.59 | 89.31 | 84.37 | 73.71 | 87.71 | — | — | — | |
| XAttentionBackbone=Qwen-2.5-7B-1M2025.10 | 92.1 | 93.69 | 91.5 | 88.35 | 81.26 | 73.05 | 86.66 | — | — | — | |
| FLEXBackbone=QWEN3-8B2026.03 | 91.98 | 95.06 | 91.17 | 90.93 | 77.31 | 70.28 | 86.12 | 30 | — | — | |
| MinferenceBackbone=Qwen-2.5-7B-1M2025.10 | 91.3 | 94.01 | 91.6 | 89.09 | 81.3 | 70.1 | 86.23 | — | — | — | |
| FullKVModel=Llama-3-8B-262K2025.12 | 90.97 | — | 90.1 | 86.17 | 83.06 | 79.65 | 85.99 | — | — | — | |
| PBS-AttnBackbone=Qwen-2.5-7B-1M2025.10 | 90.77 | 93.27 | 90.54 | 85.54 | 81.5 | 70.61 | 85.37 | — | — | — | |
| FlatModel=Yi-9B-200K, Sparsity Budget (p')=10242025.12 | 90.3 | — | 81.72 | 71.67 | 65.02 | 58.82 | -0.51 | — | — | — | |
| CTkvrModel=Yi-9B-200K, Sparsity Budget (p')=5122025.12 | 90.16 | — | 81.45 | 72.57 | 64.09 | 57.03 | -0.92 | — | — | — | |
| CTkvrModel=Yi-9B-200K, Sparsity Budget (p')=10242025.12 | 90.16 | — | 81.45 | 72.5 | 64.54 | 58.55 | -0.56 | — | — | — | |
| CTkvrModel=Llama-3-8B-262K, Sparsity Budget (p')=10242025.12 | 90.14 | — | 89.93 | 86.2 | 83.38 | 76.6 | -0.74 | — | — | — | |
| FlatModel=Llama-3-8B-262K, Sparsity Budget (p')=10242025.12 | 90.1 | — | 89.89 | 86 | 83.37 | 77.34 | -0.65 | — | — | — | |
| ShadowKVModel=Llama-3-8B-262K, Sparsity Budget (p')=10242025.12 | 90.06 | — | 88.81 | 85.74 | 80.4 | 74.39 | -2.11 | — | — | — | |
| CTkvrModel=Llama-3-8B-262K, Sparsity Budget (p')=5122025.12 | 89.9 | — | 89.65 | 86.42 | 82.71 | 74.82 | -1.29 | — | — | — | |
| FullKVModel=Yi-9B-200K2025.12 | 89.6 | — | 81.94 | 71.5 | 66.09 | 60.91 | 74.01 | — | — | — | |
| Long-SFTModel Scale=7B, Training Strategy=Long-SFT2026.04 | 89.13 | 92.16 | 87.59 | 82.58 | 76.01 | 63.78 | 81.88 | — | — | — | |
| QuestModel=Llama-3-8B-262K, Sparsity Budget (p')=10242025.12 | 88.81 | — | 86.01 | 81.16 | 75.63 | 70.48 | -5.58 | — | — | — | |
| FlexPrefillBackbone=Qwen-2.5-7B-1M2025.10 | 87.74 | 84.17 | 86.73 | 84.21 | 78.15 | 61.66 | 80.44 | — | — | — | |
| MeanPoolingBackbone=Qwen-2.5-7B-1M2025.10 | 87.43 | 90.15 | 86.38 | 82.7 | 78.86 | 67.51 | 82.17 | — | — | — | |
| ShadowKVModel=Yi-9B-200K, Sparsity Budget (p')=10242025.12 | 86.96 | — | 79.25 | 68.19 | 65.12 | 57.69 | -2.57 | — | — | — | |
| MagicPIGModel=Yi-9B-200K, Sparsity Budget (p')=10242025.12 | 86.53 | — | 78.66 | 67.56 | 62.89 | 58.54 | -3.17 | — | — | — | |
| MagicPIGModel=Llama-3-8B-262K, Sparsity Budget (p')=10242025.12 | 85.04 | — | 85 | 80.34 | 75.27 | 67.51 | -7.36 | — | — | — | |
| FullBackbone Model=LLaMA-3.1-8B-Instruct, KV Cache Management Method=Full Attention2026.06 | 85 | — | 78.6 | 76.8 | 78.2 | — | 79.65 | — | — | — | |
| QuestModel=Yi-9B-200K, Sparsity Budget (p')=10242025.12 | 84.85 | — | 75.77 | 63.23 | 58.17 | 52.22 | -7.16 | — | — | — | |
| RaBitQCacheBackbone Model=LLaMA-3.1-8B-Instruct, p-value (Adaptive Top-p)=0.92026.06 | 84.7 | — | 80.3 | 76.7 | 76.5 | — | 79.55 | — | — | 2,274 | |
| QuestBackbone Model=LLaMA-3.1-8B-Instruct, KV Cache Token Budget=40962026.06 | 84.7 | — | 78.6 | 75.2 | 74.3 | — | 78.2 | — | — | — | |
| DSBackbone Model=LLaMA-3.1-8B-Instruct, KV Cache Token Budget=40962026.06 | 84.6 | — | 79.2 | 77.2 | 74 | — | 78.75 | — | — | — | |
| SparQBackbone Model=LLaMA-3.1-8B-Instruct, KV Cache Token Budget=Default2026.06 | 84.3 | — | 79 | 76.6 | 76.4 | — | 79.08 | — | — | — | |
| DeepSeek V2-Lite (MLA)FTL (s)=10.78, Hardware=H200 GPUs, Evaluation Context=128K2026.04 | 78.82 | 79.51 | 62.28 | 58.33 | 47.57 | 23.96 | 58.91 | — | — | — | |
| LCAFTL (s)=4.40 (2.5×), Hardware=H200 GPUs, Evaluation Context=128K2026.04 | 77.19 | 77.19 | 63.75 | 59.69 | 50.63 | 24.38 | 58.8 | — | — | — | |
| KDAFTL (s)=4.96 (2.2×), Hardware=H200 GPUs, Evaluation Context=128K2026.04 | 73.61 | 72.22 | 64.24 | 51.39 | 44.1 | 22.22 | 54.63 | — | — | — | |
| FlexPrefillFTL (s)=5.38 (2.0×), Hardware=H200 GPUs, Evaluation Context=128K2026.04 | 70 | 72.81 | 56.25 | 19.06 | 9.38 | 7.19 | 39.11 | — | — | — | |
| MInferenceFTL (s)=5.66 (1.9×), Hardware=H200 GPUs, Evaluation Context=128K2026.04 | 69.69 | 76.88 | 44.06 | 20.31 | 10.31 | 4.34 | 37.6 | — | — | — | |
| RoPE++ECModel size=776M, Training length=32k2025.12 | 38.6 | 42.7 | 33.4 | 21.7 | 10.9 | — | 29.4 | — | — | — | |
| RoPE++EHModel size=776M, Training length=32k2025.12 | 35.4 | 38.7 | 33.8 | 24.6 | 10.7 | — | 28.6 | — | — | — | |
| RoPEModel size=776M, Training length=32k2025.12 | 35.1 | 37.4 | 33 | 21.2 | 10.4 | — | 27.4 | — | — | — | |
| RoPE++ECModel size=376M, Training length=32k2025.12 | 33 | 36.1 | 29.1 | 17.7 | 9 | — | 25 | — | — | — | |
| RoPE++EHModel size=376M, Training length=32k2025.12 | 28.4 | 29.9 | 17.6 | 9.4 | 5.9 | — | 18.2 | — | — | — | |
| RoPEModel size=376M, Training length=32k2025.12 | 25.6 | 31.6 | 22 | 9.5 | 5.5 | — | 18.8 | — | — | — | |
| SnapKVModel=Llama-3-8B-262K, Sparsity Budget (p')=10242025.12 | 21.17 | — | 16.11 | 5.9 | 2.58 | 1 | -76.64 | — | — | — | |
| SnapKVModel=Yi-9B-200K, Sparsity Budget (p')=10242025.12 | 7.77 | — | 9.03 | 7.1 | 7.05 | 2.83 | -67.25 | — | — | — | |
| PyramidInferBackbone=Llama-3.1-8B-UL, KV Budget=20482026.01 | 3.38 | 2.09 | — | — | — | — | — | — | — | — | |
| PyramidInferBackbone=Qwen2.5-7B, KV Budget=20482026.01 | 3.15 | 1.94 | — | — | — | — | — | — | — | — | |
| ASL_2passBackbone=Qwen2.5-7B, KV Budget=20482026.01 | 1.43 | 1.73 | 1.25 | 1.48 | 0.79 | 0.95 | — | — | — | — | |
| ASL_2passBackbone=Llama-3.1-8B-UL, KV Budget=20482026.01 | 1.17 | 1.65 | 0.92 | 0.65 | 0.69 | 0.8 | — | — | — | — | |
| SnapKVBackbone=Llama-3.1-8B-UL, KV Budget=20482026.01 | 1.05 | 1.08 | 1.04 | 1.03 | 1.01 | 1.01 | — | — | — | — | |
| SnapKVBackbone=Qwen2.5-7B, KV Budget=20482026.01 | 1.04 | 1.07 | 1.03 | 1.02 | 1.01 | 1.01 | — | — | — | — | |
| ASLBackbone=Qwen2.5-7B, KV Budget=20482026.01 | 1.03 | 1.12 | 0.93 | 0.83 | 0.79 | 0.81 | — | — | — | — | |
| ASLBackbone=Llama-3.1-8B-UL, KV Budget=20482026.01 | 0.92 | 1.06 | 0.82 | 0.61 | 0.67 | 0.79 | — | — | — | — | |
| GemFilterBackbone=Qwen2.5-7B, KV Budget=20482026.01 | 0.83 | 1.19 | 0.66 | 0.57 | 0.55 | 0.53 | — | — | — | — | |
| GemFilterBackbone=Llama-3.1-8B-UL, KV Budget=20482026.01 | 0.74 | 1.11 | 0.57 | 0.49 | 0.45 | 0.44 | — | — | — | — | |
| FastKVBackbone=Qwen2.5-7B, KV Budget=20482026.01 | 0.69 | 0.87 | 0.61 | 0.56 | 0.55 | 0.54 | — | — | — | — | |
| FastKVBackbone=Llama-3.1-8B-UL, KV Budget=20482026.01 | 0.66 | 0.85 | 0.58 | 0.53 | 0.51 | 0.5 | — | — | — | — | |
| Full (exact)Model Backbone=LLAMA-3.2-1B-INSTRUCT, Load Budget=100% (Full)2026.04 | — | — | — | — | — | — | — | — | 80.5 | — | |
| Full (exact)Model Backbone=QWEN3-1.7B, Load Budget=100% (Full)2026.04 | — | — | — | — | — | — | — | — | 89.1 | — | |
| MAGICPIGModel Backbone=LLAMA-3.2-1B-INSTRUCT, Load Budget=2%2026.04 | — | — | — | — | — | — | — | — | 60 | — | |
| MAGICPIGModel Backbone=LLAMA-3.2-1B-INSTRUCT, Load Budget=3%2026.04 | — | — | — | — | — | — | — | — | 72.1 | — | |
| MAGICPIGModel Backbone=LLAMA-3.2-1B-INSTRUCT, Load Budget=4%2026.04 | — | — | — | — | — | — | — | — | 69.9 | — | |
| MAGICPIGModel Backbone=LLAMA-3.2-1B-INSTRUCT, Load Budget=5%2026.04 | — | — | — | — | — | — | — | — | 68.1 | — | |
| MAGICPIGModel Backbone=QWEN3-1.7B, Load Budget=3%2026.04 | — | — | — | — | — | — | — | — | 84.6 | — | |
| MAGICPIGModel Backbone=QWEN3-1.7B, Load Budget=4%2026.04 | — | — | — | — | — | — | — | — | 84.2 | — | |
| MAGICPIGModel Backbone=QWEN3-1.7B, Load Budget=5%2026.04 | — | — | — | — | — | — | — | — | 83.3 | — | |
| PYRAMIDKVModel Backbone=LLAMA-3.2-1B-INSTRUCT, Load Budget=2%2026.04 | — | — | — | — | — | — | — | — | 6 | — | |
| PYRAMIDKVModel Backbone=LLAMA-3.2-1B-INSTRUCT, Load Budget=3%2026.04 | — | — | — | — | — | — | — | — | 10.8 | — | |
| PYRAMIDKVModel Backbone=LLAMA-3.2-1B-INSTRUCT, Load Budget=4%2026.04 | — | — | — | — | — | — | — | — | 15.7 | — | |
| PYRAMIDKVModel Backbone=LLAMA-3.2-1B-INSTRUCT, Load Budget=5%2026.04 | — | — | — | — | — | — | — | — | 23.9 | — | |
| PYRAMIDKVModel Backbone=QWEN3-1.7B, Load Budget=3%2026.04 | — | — | — | — | — | — | — | — | 22.6 | — | |
| PYRAMIDKVModel Backbone=QWEN3-1.7B, Load Budget=4%2026.04 | — | — | — | — | — | — | — | — | 22.6 | — | |
| PYRAMIDKVModel Backbone=QWEN3-1.7B, Load Budget=5%2026.04 | — | — | — | — | — | — | — | — | 32 | — | |
| QUESTModel Backbone=LLAMA-3.2-1B-INSTRUCT, Load Budget=2%2026.04 | — | — | — | — | — | — | — | — | 39 | — | |
| QUESTModel Backbone=LLAMA-3.2-1B-INSTRUCT, Load Budget=3%2026.04 | — | — | — | — | — | — | — | — | 47.7 | — | |
| QUESTModel Backbone=LLAMA-3.2-1B-INSTRUCT, Load Budget=4%2026.04 | — | — | — | — | — | — | — | — | 51.2 | — | |
| QUESTModel Backbone=LLAMA-3.2-1B-INSTRUCT, Load Budget=5%2026.04 | — | — | — | — | — | — | — | — | 54.1 | — |