Long-context evaluation on RULER 32k
90.06Overall ScoreRTPurbo
Evaluation results
| Method | Links | |||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| RTPurboselection_mode=top-p2026.05 | 90.06 | — | — | — | — | — | — | — | — | 97.84 | 85.44 | 91.33 | 100 | — | — | — | — | — | 64.6 | 72.63 | 99.95 | 99.15 | 99.6 | |
| Full Attn2026.05 | 89.65 | — | — | — | — | — | — | — | — | 98 | 81.8 | 91.47 | 99.93 | — | — | — | — | — | 64.6 | 72.63 | 99.95 | 98.85 | 99.66 | |
| HySparseModel Size=7B2026.02 | 89.3 | 100 | 100 | 99.8 | 98.2 | 99.6 | 76.2 | 88.8 | 94.7 | 91.2 | 38.8 | 95.1 | — | — | — | — | — | — | — | — | — | — | — | |
| RazorAttn2026.05 | 88.69 | — | — | — | — | — | — | — | — | 92.04 | 82.5 | 90.07 | 99.8 | — | — | — | — | — | 65 | 71.88 | 99.6 | 97.65 | 99.66 | |
| Full-AttnModel Size=7B2026.02 | 88.2 | 100 | 100 | 100 | 98 | 99.4 | 75.8 | 96.4 | 98.3 | 90.5 | 16.6 | 95.1 | — | — | — | — | — | — | — | — | — | — | — | |
| HySparseModel Size=80B2026.02 | 87.4 | 100 | 100 | 99.4 | 96.8 | 99 | 98.4 | 89.5 | 85.7 | 89.6 | 20.8 | 82.1 | — | — | — | — | — | — | — | — | — | — | — | |
| RTPurboselection_mode=top-k2026.05 | 84.36 | — | — | — | — | — | — | — | — | 97.68 | 82.5 | 91 | 94.06 | — | — | — | — | — | 58.4 | 71.78 | 99.7 | 98.55 | 65.53 | |
| Hybrid SWAModel Size=7B2026.02 | 84.2 | 100 | 100 | 99.6 | 96.2 | 98.8 | 53.4 | 93.4 | 87.2 | 98.5 | 10.4 | 88.5 | — | — | — | — | — | — | — | — | — | — | — | |
| Minference2026.05 | 83.58 | — | — | — | — | — | — | — | — | 95.84 | 80.88 | 85.33 | 98.8 | — | — | — | — | — | 62 | 69.88 | 97.45 | 91.35 | 70.73 | |
| SnapKV2026.05 | 83.43 | — | — | — | — | — | — | — | — | 94.23 | 76.23 | 89.92 | 99.86 | — | — | — | — | — | 58.92 | 69.2 | 94 | 71.1 | 97.46 | |
| FlexPrefill2026.05 | 83.4 | — | — | — | — | — | — | — | — | 96.56 | 75.26 | 90 | 100 | — | — | — | — | — | 60.4 | 71.18 | 98.2 | 60.3 | 98.66 | |
| Full-AttnModel Size=80B2026.02 | 82.1 | 100 | 99.2 | 81.2 | 99 | 99.4 | 77 | 86 | 79.5 | 74.5 | 40.7 | 66.7 | — | — | — | — | — | — | — | — | — | — | — | |
| Quest2026.05 | 78.97 | — | — | — | — | — | — | — | — | 97.48 | 57.26 | 82.6 | 90.27 | — | — | — | — | — | 56.8 | 68.73 | 85.6 | 85.2 | 86.8 | |
| QUOKABackbone=Qwen-3-4B, BSA (Budget Size of Attention)=1,0242026.02 | 74.83 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Hybrid SWAModel Size=80B2026.02 | 69.5 | 100 | 98.2 | 58.8 | 89.6 | 90.2 | 61 | 74.1 | 56.6 | 73.1 | 8.4 | 54.3 | — | — | — | — | — | — | — | — | — | — | — | |
| QUOKABackbone=Smollm3, BSA (Budget Size of Attention)=1,0242026.02 | 61.37 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| QUOKABackbone=Qwen-2.5-3B-Instruct, BSA (Budget Size of Attention)=1,0242026.02 | 59.37 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| QUOKABackbone=GPT-OSS-20B, BSA (Budget Size of Attention)=1,0242026.02 | 57.79 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| QUOKABackbone=Llama-3.2-3B-Instruct, BSA (Budget Size of Attention)=1,0242026.02 | 57.01 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| SampleAttnBackbone=Smollm3, BSA (Budget Size of Attention)=1,0242026.02 | 45.98 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| SampleAttnBackbone=Qwen-3-4B, BSA (Budget Size of Attention)=1,0242026.02 | 40.72 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| LokiBackbone=GPT-OSS-20B, BSA (Budget Size of Attention)=1,0242026.02 | 39.92 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| LokiBackbone=Qwen-3-4B, BSA (Budget Size of Attention)=1,0242026.02 | 39.31 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| SparQBackbone=Qwen-2.5-3B-Instruct, BSA (Budget Size of Attention)=1,0242026.02 | 36.74 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| SampleAttnBackbone=Qwen-2.5-3B-Instruct, BSA (Budget Size of Attention)=1,0242026.02 | 36.17 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| SparQBackbone=Qwen-3-4B, BSA (Budget Size of Attention)=1,0242026.02 | 35.2 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| LokiBackbone=Qwen-2.5-3B-Instruct, BSA (Budget Size of Attention)=1,0242026.02 | 34.12 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| SampleAttnBackbone=Llama-3.2-3B-Instruct, BSA (Budget Size of Attention)=1,0242026.02 | 31.73 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| SparQBackbone=Llama-3.2-3B-Instruct, BSA (Budget Size of Attention)=1,0242026.02 | 31.14 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| SampleAttnBackbone=GPT-OSS-20B, BSA (Budget Size of Attention)=1,0242026.02 | 30.42 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| KeyDiffBackbone=Smollm3, BSA (Budget Size of Attention)=1,0242026.02 | 30.37 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| LessIsMoreBackbone=Smollm3, BSA (Budget Size of Attention)=1,0242026.02 | 24.21 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| LokiBackbone=Smollm3, BSA (Budget Size of Attention)=1,0242026.02 | 22.66 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| LessIsMoreBackbone=GPT-OSS-20B, BSA (Budget Size of Attention)=1,0242026.02 | 20.11 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| LessIsMoreBackbone=Llama-3.2-3B-Instruct, BSA (Budget Size of Attention)=1,0242026.02 | 19.16 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| SparQBackbone=Smollm3, BSA (Budget Size of Attention)=1,0242026.02 | 18.69 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| SnapKVBackbone=GPT-OSS-20B, BSA (Budget Size of Attention)=1,0242026.02 | 16.63 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| SparQBackbone=GPT-OSS-20B, BSA (Budget Size of Attention)=1,0242026.02 | 15.2 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| KeyDiffBackbone=Llama-3.2-3B-Instruct, BSA (Budget Size of Attention)=1,0242026.02 | 14.87 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| LessIsMoreBackbone=Qwen-3-4B, BSA (Budget Size of Attention)=1,0242026.02 | 14.87 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| SnapKVBackbone=Smollm3, BSA (Budget Size of Attention)=1,0242026.02 | 12.23 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| KeyDiffBackbone=Qwen-3-4B, BSA (Budget Size of Attention)=1,0242026.02 | 12.07 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| KeyDiffBackbone=Qwen-2.5-3B-Instruct, BSA (Budget Size of Attention)=1,0242026.02 | 10.32 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| LessIsMoreBackbone=Qwen-2.5-3B-Instruct, BSA (Budget Size of Attention)=1,0242026.02 | 10.12 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| SnapKVBackbone=Qwen-3-4B, BSA (Budget Size of Attention)=1,0242026.02 | 10.07 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| KeyDiffBackbone=GPT-OSS-20B, BSA (Budget Size of Attention)=1,0242026.02 | 9.65 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| SnapKVBackbone=Qwen-2.5-3B-Instruct, BSA (Budget Size of Attention)=1,0242026.02 | 8.13 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| LokiBackbone=Llama-3.2-3B-Instruct, BSA (Budget Size of Attention)=1,0242026.02 | 8.05 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| SnapKVBackbone=Llama-3.2-3B-Instruct, BSA (Budget Size of Attention)=1,0242026.02 | 6.21 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Engram-27BPre-training Steps=41k, Pre-training Loss=1.66, Context Window Size=32k2026.01 | — | — | — | — | — | — | — | — | — | 83.2 | 3.8 | 99.6 | 99.6 | 88.3 | 93 | 89.5 | 44 | — | — | — | — | — | — | |
| Engram-27BPre-training Steps=46k, Pre-training Loss=1.63, Context Window Size=32k2026.01 | — | — | — | — | — | — | — | — | — | 87.2 | 4.3 | 98.6 | 97.6 | 89 | 95.5 | 97 | 37.5 | — | — | — | — | — | — | |
| Engram-27BPre-training Steps=50k, Pre-training Loss=1.62, Context Window Size=32k2026.01 | — | — | — | — | — | — | — | — | — | 89 | 5.9 | 99.3 | 99.3 | 89.3 | 96.5 | 97 | 40.5 | — | — | — | — | — | — | |
| Gemma3-4BModel Size=4B2026.04 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 43.44 | — | — | — | — | — | |
| Llama3.2-3BModel Size=3B2026.04 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 51.38 | — | — | — | — | — | |
| MoE-27BPre-training Steps=50k, Pre-training Loss=1.63, Context Window Size=32k2026.01 | — | — | — | — | — | — | — | — | — | 77 | 4.5 | 73 | 100 | 88 | 92.7 | 84.2 | 34.5 | — | — | — | — | — | — | |
| Qwen2.5-3BModel Size=3B2026.04 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 58.95 | — | — | — | — | — | |
| Qwen3-4B-BaseModel Size=4B, Training Strategy=Base2026.04 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 62.28 | — | — | — | — | — | |
| Qwen3-4B-Instruct-2507Model Size=4B, Training Strategy=Instruct-25072026.04 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 85.62 | — | — | — | — | — | |
| SpB1.0-7BModel Size=7B2026.04 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 17.21 | — | — | — | — | — | |
| SpB2.0-5BModel Size=5B2026.04 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 50.87 | — | — | — | — | — |