Long-context understanding on RULER (4K to 64K Contexts)
97.3AccuracyGemma 4 26B-A4B
Evaluation results
| Method | Links | |||||||
|---|---|---|---|---|---|---|---|---|
| Gemma 4 26B-A4BModel Size/Architecture=26B-A4B, Context length=32k, Thinking mode=without thinking2026.07 | 97.3 | — | — | — | — | — | — | |
| Gemma 4 31BModel Size/Architecture=31B, Context length=32k, Thinking mode=without thinking2026.07 | 96.8 | — | — | — | — | — | — | |
| Gemma 4 31BModel Size/Architecture=31B, Context length=128k, Thinking mode=without thinking2026.07 | 96.4 | — | — | — | — | — | — | |
| Gemma 4 12BModel Size/Architecture=12B, Context length=32k, Thinking mode=without thinking2026.07 | 96.4 | — | — | — | — | — | — | |
| Gemma 4 E4BModel Size/Architecture=E4B, Context length=32k, Thinking mode=without thinking2026.07 | 95.2 | — | — | — | — | — | — | |
| PulseColModel=LLaDA-1.5, Attention Sparsity=80%2026.05 | 91.72 | — | — | — | — | — | 66.58 | |
| Gemma 4 12BModel Size/Architecture=12B, Context length=128k, Thinking mode=without thinking2026.07 | 91.2 | — | — | — | — | — | — | |
| Gemma 3 27BModel Size/Architecture=27B, Context length=32k, Thinking mode=without thinking2026.07 | 91.1 | — | — | — | — | — | — | |
| SparseDModel=LLaDA-1.5, Attention Sparsity=80%2026.05 | 90.95 | — | — | — | — | — | 63.89 | |
| Full AttentionModel=LLaDA-1.52026.05 | 90.69 | — | — | — | — | — | 67.58 | |
| SparseDModel=LLaDA-1.5, Attention Sparsity=50%2026.05 | 90.62 | — | — | — | — | — | 67.27 | |
| DenseBackbone=Qwen3-30B-A3B-Instruct-2507, Chunk Size=10242026.05 | 90.46 | — | — | — | — | — | — | |
| PulseColModel=LLaDA-1.5, Attention Sparsity=50%2026.05 | 90.36 | — | — | — | — | — | 67.47 | |
| Full AttentionModel=Dream2026.05 | 89.9 | — | — | — | — | — | 73.56 | |
| SparseDModel=Dream, Attention Sparsity=50%2026.05 | 89.83 | — | — | — | — | — | 70.85 | |
| PulseColModel=Dream, Attention Sparsity=50%2026.05 | 89.82 | — | — | — | — | — | 74.08 | |
| Gemma 4 26B-A4BModel Size/Architecture=26B-A4B, Context length=128k, Thinking mode=without thinking2026.07 | 89.8 | — | — | — | — | — | — | |
| XAttentionBackbone=Qwen3-30B-A3B-Instruct-2507, Chunk Size=10242026.05 | 89.78 | — | — | — | — | — | — | |
| PulseColModel=Dream, Attention Sparsity=80%2026.05 | 89.4 | — | — | — | — | — | 72.83 | |
| CompactAttention-FPBackbone=Qwen3-30B-A3B-Instruct-2507, Chunk Size=10242026.05 | 89.29 | — | — | — | — | — | — | |
| FlashPrefillBackbone=Qwen3-30B-A3B-Instruct-2507, Chunk Size=10242026.05 | 88.68 | — | — | — | — | — | — | |
| Gemma 4 E4BModel Size/Architecture=E4B, Context length=128k, Thinking mode=without thinking2026.07 | 86.6 | — | — | — | — | — | — | |
| Gemma 4 E2BModel Size/Architecture=E2B, Context length=32k, Thinking mode=without thinking2026.07 | 83 | — | — | — | — | — | — | |
| DenseBackbone=LLaMA-3.1-8B-Instruct, Chunk Size=10242026.05 | 82.78 | — | — | — | — | — | — | |
| SeerAttentionBackbone=LLaMA-3.1-8B-Instruct, Chunk Size=10242026.05 | 82.46 | — | — | — | — | — | — | |
| CompactAttention-SABackbone=LLaMA-3.1-8B-Instruct, Chunk Size=10242026.05 | 82.44 | — | — | — | — | — | — | |
| FlashPrefillBackbone=LLaMA-3.1-8B-Instruct, Chunk Size=10242026.05 | 82.35 | — | — | — | — | — | — | |
| XAttentionBackbone=LLaMA-3.1-8B-Instruct, Chunk Size=10242026.05 | 82.31 | — | — | — | — | — | — | |
| CompactAttention-FPBackbone=LLaMA-3.1-8B-Instruct, Chunk Size=10242026.05 | 82.3 | — | — | — | — | — | — | |
| SparseDModel=Dream, Attention Sparsity=80%2026.05 | 81.55 | — | — | — | — | — | 49.85 | |
| QUOKABackbone=Qwen3-30B-A3B-Instruct-2507, Chunk Size=10242026.05 | 80.19 | — | — | — | — | — | — | |
| QUOKABackbone=LLaMA-3.1-8B-Instruct, Chunk Size=10242026.05 | 77.98 | — | — | — | — | — | — | |
| Gemma 4 E2BModel Size/Architecture=E2B, Context length=128k, Thinking mode=without thinking2026.07 | 70.4 | — | — | — | — | — | — | |
| Gemma 3 27BModel Size/Architecture=27B, Context length=128k, Thinking mode=without thinking2026.07 | 66 | — | — | — | — | — | — | |
| StreamingLLMModel=Dream2026.05 | 50.52 | — | — | — | — | — | 53.23 | |
| Sliding WindowModel=Dream2026.05 | 46.74 | — | — | — | — | — | 48.49 | |
| StreamingLLMModel=LLaDA-1.52026.05 | 45.02 | — | — | — | — | — | 44.9 | |
| Sliding WindowModel=LLaDA-1.52026.05 | 43.35 | — | — | — | — | — | 39.58 | |
| AdaKV w/ CriticalKVBackbone=Llama-3.1-8B, Cache Budget=40%2025.02 | — | — | — | — | — | — | 86.28 | |
| DenseModel=Llama-3.1-8B, Evaluation Phase=Prefill2026.02 | — | 95.31 | 95.31 | 94.06 | 89.69 | 82.81 | 91.44 | |
| DenseModel=Llama-3.2-1B, Evaluation Phase=Prefill2026.02 | — | 74.38 | 68.44 | 65.31 | 66.88 | 66.56 | 68.31 | |
| FlexPrefillModel=Llama-3.1-8B, Evaluation Phase=Prefill, Sparsity=80%2026.02 | — | 95.31 | 92.5 | 93.75 | 88.13 | 80.31 | 90 | |
| FlexPrefillModel=Llama-3.2-1B, Evaluation Phase=Prefill, Sparsity=80%2026.02 | — | 74.06 | 63.75 | 60.63 | 61.88 | 55.94 | 63.25 | |
| HeadKV w/ CriticalKVBackbone=Llama-3.1-8B, Cache Budget=40%2025.02 | — | — | — | — | — | — | 89.29 | |
| MInferenceModel=Llama-3.1-8B, Evaluation Phase=Prefill, Sparsity=80%2026.02 | — | 92.65 | 95 | 93.12 | 86.87 | 82.5 | 90.03 | |
| MInferenceModel=Llama-3.2-1B, Evaluation Phase=Prefill, Sparsity=80%2026.02 | — | 73.44 | 69.06 | 64.69 | 64.38 | 62.71 | 66.86 | |
| Sketch&WalkModel=Llama-3.1-8B, Evaluation Phase=Prefill, Sparsity=80%, Sketching Dimension=64, Sparsity Exponent=82026.02 | — | 96.56 | 94.06 | 94.06 | 89.63 | 82.5 | 91.35 | |
| Sketch&WalkModel=Llama-3.2-1B, Evaluation Phase=Prefill, Sparsity=80%, Sketching Dimension=64, Sparsity Exponent=82026.02 | — | 74.06 | 68.44 | 66.06 | 65.62 | 63.75 | 67.59 | |
| SnapKVBackbone=Llama-3.1-8B, Cache Budget=40%2025.02 | — | — | — | — | — | — | 67.93 | |
| SnapKV w/ CriticalKVBackbone=Llama-3.1-8B, Cache Budget=40%2025.02 | — | — | — | — | — | — | 76.89 | |
| SnapKV w/ OBCache (Joint)Backbone=Llama-3.1-8B, Cache Budget=40%2025.02 | — | — | — | — | — | — | 71.53 |