Long-context evaluation on RULER
92.8Average Accuracy ScoreRecaLLM-Qwen2.5-7B
Evaluation results
| Method | Links | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| RecaLLM-Qwen2.5-7B2026.04 | 92.8 | 98.8 | 96.7 | 95.8 | 93.1 | 89.7 | 82.9 | — | |
| LLaMA3.1-8BBackbone=LLaMA3.1-8B, Configuration=Full-cache baseline2026.03 | 92.78 | 94.74 | 92.72 | 93.51 | 90.16 | — | — | — | |
| LoongRL-14B2026.04 | 91.9 | 97.6 | 96.1 | 95.4 | 95.1 | 87.1 | 79.9 | — | |
| Llama-3.1Backbone=Llama-3.1-8B-Instruct, Sparsity=Dense2025.07 | 90.4 | 96.6 | 95.3 | 94.8 | 91.3 | 86.3 | 78.1 | — | |
| TriangleMixBackbone=Llama-3.1-8B-Instruct, Sparsity=TriangleMix2025.07 | 90.2 | 96.3 | 95.1 | 94.7 | 91.3 | 86.3 | 77.5 | — | |
| LoongRL-7B2026.04 | 89.5 | 95.1 | 94.3 | 93.4 | 91.4 | 86.2 | 76.8 | — | |
| Llama 3.1-8B (base)Precision=full precision2025.10 | 88.3 | — | — | — | — | — | — | — | |
| RecaLLM-Llama-3.1-8B2026.04 | 87.7 | 97.7 | 97.2 | 95.5 | 90.7 | 78.6 | 66.4 | — | |
| VQKVBackbone=LLaMA3.1-8B2026.03 | 87.31 | 92.88 | 89.61 | 87.02 | 79.71 | — | — | — | |
| TriangleMixBackbone=Llama-3-8B-Instruct-262K, Sparsity=TriangleMix2025.07 | 86.6 | 93.5 | 91 | 88.1 | 85 | 82.4 | 79.6 | — | |
| Llama-3-262kBackbone=Llama-3-8B-Instruct-262K, Sparsity=Dense2025.07 | 86.5 | 93.4 | 90.3 | 88.8 | 85.1 | 82.2 | 79.4 | — | |
| DuoAttentionBackbone=Llama-3-8B-Instruct-262K, Sparsity=DuoAttention2025.07 | 86 | 92.8 | 91.9 | 89 | 85 | 81.1 | 76.1 | — | |
| Qwen2.5Backbone=Qwen2.5-7B-Instruct, Sparsity=Dense2025.07 | 86 | 95.8 | 93.6 | 92.6 | 84.5 | 81.9 | 67.4 | — | |
| StreamingMixBackbone=Llama-3.1-8B-Instruct, Sparsity=StreamingMix2025.07 | 85.6 | 94.3 | 91.8 | 90.2 | 86.2 | 79.2 | 71.9 | — | |
| TriangleMixBackbone=Qwen2.5-7B-Instruct, Sparsity=TriangleMix2025.07 | 85.6 | 95.5 | 93.8 | 92.1 | 84.6 | 80.2 | 67.7 | — | |
| FlashAttnModel=LLaMA-3.1-8B-Instruct2026.03 | 85.43 | 96.48 | 94.7 | 89.84 | 79.82 | 77.48 | 74.27 | — | |
| VSPrefillModel=LLaMA-3.1-8B-Instruct2026.03 | 84.7 | 96.67 | 94.5 | 90.68 | 78.26 | 75.34 | 72.76 | 1.75 | |
| DuoAttentionBackbone=Llama-3.1-8B-Instruct, Sparsity=DuoAttention2025.07 | 84.7 | 95.7 | 93.1 | 88.4 | 84.3 | 82.5 | 64 | — | |
| LLaMA3.2-3BBackbone=LLaMA3.2-3B, Configuration=Full-cache baseline2026.03 | 84.38 | 90.04 | 85.91 | 83.3 | 78.25 | — | — | — | |
| QwenLong-L1-32BContext Length=32K2026.04 | 84.2 | 91.7 | 88.5 | 87.6 | 86.8 | 80.6 | 70.2 | — | |
| FlexPreModel=LLaMA-3.1-8B-Instruct2026.03 | 84.01 | 93.19 | 93.41 | 90.83 | 77.58 | 75.27 | 73.8 | 1.69 | |
| SeerAttnModel=LLaMA-3.1-8B-Instruct2026.03 | 84 | 93.83 | 93.83 | 90 | 78.86 | 75.19 | 72.29 | 1.59 | |
| StreamingMixBackbone=Qwen2.5-7B-Instruct, Sparsity=StreamingMix2025.07 | 82.5 | 93.3 | 90.4 | 89.4 | 81.7 | 76.4 | 63.6 | — | |
| Qwen2.5-7B-Instruct2026.04 | 81.8 | 92 | 86.8 | 86.1 | 82.3 | 76.8 | 66.8 | — | |
| Llama-3.1-8B-Instruct2026.04 | 80.9 | 95.7 | 93.5 | 89.8 | 85.6 | 78.6 | 42.2 | — | |
| StreamingMixBackbone=Llama-3-8B-Instruct-262K, Sparsity=StreamingMix2025.07 | 79.9 | 87.3 | 83.2 | 80.8 | 79.9 | 77.6 | 70.8 | — | |
| FlashAttnModel=Qwen3-4B-Instruct2026.03 | 79.67 | 77.99 | 79.4 | 80.23 | 78.96 | 84 | 77.41 | — | |
| VSPrefillModel=Qwen3-4B-Instruct2026.03 | 78.61 | 77.46 | 77.27 | 80 | 78.29 | 83.47 | 75.15 | 1.91 | |
| VQKVBackbone=LLaMA3.2-3B2026.03 | 77.11 | 88.24 | 78.72 | 73.58 | 67.89 | — | — | — | |
| FlexPreModel=Qwen3-4B-Instruct2026.03 | 76.43 | 74.32 | 76.62 | 76.87 | 76.87 | 79.44 | 74.44 | 1.83 | |
| SeerAttnModel=Qwen3-4B-Instruct2026.03 | 76.38 | 77.1 | 78.24 | 79.92 | 77.16 | 81.43 | 64.45 | 1.67 | |
| Qwen3-1.7B + SFT + dGRPOModel Size=1.7B, Training=SFT + dGRPO2026.05 | 74.8 | 88.3 | 85.4 | 84.7 | 80.4 | 65.6 | 44.5 | — | |
| SnapKVBackbone=LLaMA3.1-8B, recalled tokens=20482026.03 | 74.16 | 91.18 | 77.12 | 69.8 | 58.52 | — | — | — | |
| TriangleBackbone=Llama-3-8B-Instruct-262K, Sparsity=Triangle2025.07 | 74.1 | 88.1 | 84.7 | 80.6 | 71 | 65 | 55.4 | — | |
| CAGEBase Model=Llama 3.1-8B, Quantization configuration (W/A bits)=W4A16, Fine-tuning protocol=fine-tuning on TULU-SFT2025.10 | 73.2 | — | — | — | — | — | — | — | |
| TriangleBackbone=Llama-3.1-8B-Instruct, Sparsity=Triangle2025.07 | 72.7 | 88.8 | 88.3 | 82.8 | 72.6 | 65 | 39 | — | |
| ProLong-8B-Instruct-512KContext Length=512K2026.04 | 69.8 | 92.3 | 86 | 73.7 | 59.5 | 60.5 | 46.8 | — | |
| SnapKVBackbone=LLaMA3.2-3B, recalled tokens=20482026.03 | 68.84 | 85.33 | 69.57 | 61.73 | 58.74 | — | — | — | |
| QuESTBase Model=Llama 3.1-8B, Quantization configuration (W/A bits)=W4A16, Fine-tuning protocol=fine-tuning on TULU-SFT2025.10 | 68.7 | — | — | — | — | — | — | — | |
| GPTQBase Model=Llama 3.1-8B, Quantization configuration (W/A bits)=W4A16, Fine-tuning protocol=fine-tuning on TULU-SFT2025.10 | 65.1 | — | — | — | — | — | — | — | |
| Qwen3-1.7BModel Size=1.7B2026.05 | 64.5 | 87.4 | 84.2 | 81 | 72 | 51.5 | 11 | — | |
| PaluBackbone=LLaMA3.1-8B, KV retention=70%2026.03 | 63.24 | 74.7 | 66.01 | 59.8 | 52.44 | — | — | — | |
| PaluBackbone=LLaMA3.2-3B, KV retention=70%2026.03 | 62.98 | 71.75 | 65.38 | 59.69 | 55.08 | — | — | — | |
| TriangleBackbone=Qwen2.5-7B-Instruct, Sparsity=Triangle2025.07 | 61.4 | 88.4 | 81.9 | 74 | 58.3 | 51 | 14.6 | — | |
| StrLLMModel=Qwen3-4B-Instruct2026.03 | 55.03 | 70.85 | 64.84 | 58.47 | 52.31 | 47.56 | 36.16 | 4.79 | |
| KIVIBackbone=LLaMA3.1-8B, quantization=4-bit, compression=75%2026.03 | 52.01 | 57.91 | 52.81 | 47.6 | 49.73 | — | — | — | |
| AdaKV w/ CriticalKVBackbone=Mistral-7B, Cache Budget=20%2025.02 | 44.13 | — | — | — | — | — | — | — | |
| RTNBase Model=Llama 3.1-8B, Quantization configuration (W/A bits)=W4A16, Fine-tuning protocol=fine-tuning on TULU-SFT2025.10 | 41.5 | — | — | — | — | — | — | — | |
| KIVIBackbone=LLaMA3.2-3B, quantization=4-bit, compression=75%2026.03 | 41.25 | 37.92 | 49.37 | 41.41 | 36.28 | — | — | — | |
| StrLLMModel=LLaMA-3.1-8B-Instruct2026.03 | 41.21 | 82.02 | 69.76 | 45.56 | 30.77 | 12.52 | 6.64 | 6.71 | |
| StreamingBackbone=Llama-3.1-8B-Instruct, Sparsity=Streaming2025.07 | 36.5 | 64.1 | 55.4 | 40.5 | 28.9 | 26.7 | 3.3 | — | |
| StreamingBackbone=Qwen2.5-7B-Instruct, Sparsity=Streaming2025.07 | 36.5 | 55.6 | 47.8 | 35.9 | 30.6 | 28.1 | 21 | — | |
| HeadKV w/ CriticalKVBackbone=Mistral-7B, Cache Budget=20%2025.02 | 34.86 | — | — | — | — | — | — | — | |
| StreamingBackbone=Llama-3-8B-Instruct-262K, Sparsity=Streaming2025.07 | 33.3 | 49.4 | 38.7 | 33.4 | 30.2 | 26.5 | 21.7 | — | |
| SnapKV w/ CriticalKVBackbone=Mistral-7B, Cache Budget=20%2025.02 | 30.24 | — | — | — | — | — | — | — | |
| ASVDBackbone=LLaMA3.1-8B, ratio=0.22026.03 | 29.31 | 39.75 | 32.94 | 25.27 | 19.29 | — | — | — | |
| SnapKV w/ OBCache (Joint)Backbone=Mistral-7B, Cache Budget=20%2025.02 | 25.99 | — | — | — | — | — | — | — | |
| SnapKVBackbone=Mistral-7B, Cache Budget=20%2025.02 | 25.6 | — | — | — | — | — | — | — | |
| ASVDBackbone=LLaMA3.2-3B, ratio=0.22026.03 | 18.36 | 27.15 | 20.83 | 15.95 | 9.51 | — | — | — | |
| Full AttentionModel Group=Baselines, Parameter Count=4B2026.04 | — | 45.77 | 38.09 | 6.58 | — | — | — | — | |
| In-Place TTTArchitecture=Full Attention, Parameter Count=4B2026.04 | — | 49.98 | 43.82 | 19.99 | — | — | — | — | |
| In-Place TTTArchitecture=Sliding-Window Attention (SWA), Parameter Count=4B2026.04 | — | 28.33 | 26.8 | 7.57 | — | — | — | — | |
| Sliding-Window Attention (SWA)Model Group=Baselines, Parameter Count=4B2026.04 | — | 14.77 | 9.91 | 5.07 | — | — | — | — |