Long-context evaluation on RULER 4k
93.73ScoreQUOKA
Evaluation results
| Method | Links | ||
|---|---|---|---|
| QUOKABackbone=Qwen-3-4B, BSA (Budget Size of Attention)=1,0242026.02 | 93.73 | — | |
| QUOKABackbone=Smollm3, BSA (Budget Size of Attention)=1,0242026.02 | 89.97 | — | |
| SparQBackbone=Qwen-3-4B, BSA (Budget Size of Attention)=1,0242026.02 | 87.93 | — | |
| QUOKABackbone=Qwen-2.5-3B-Instruct, BSA (Budget Size of Attention)=1,0242026.02 | 87.85 | — | |
| SampleAttnBackbone=Qwen-3-4B, BSA (Budget Size of Attention)=1,0242026.02 | 87.84 | — | |
| QUOKABackbone=Llama-3.2-3B-Instruct, BSA (Budget Size of Attention)=1,0242026.02 | 86.71 | — | |
| SampleAttnBackbone=Smollm3, BSA (Budget Size of Attention)=1,0242026.02 | 85.72 | — | |
| LokiBackbone=Smollm3, BSA (Budget Size of Attention)=1,0242026.02 | 84.52 | — | |
| LokiBackbone=Qwen-3-4B, BSA (Budget Size of Attention)=1,0242026.02 | 82.83 | — | |
| SparQBackbone=Smollm3, BSA (Budget Size of Attention)=1,0242026.02 | 82.45 | — | |
| LessIsMoreBackbone=Smollm3, BSA (Budget Size of Attention)=1,0242026.02 | 79.67 | — | |
| SparQBackbone=Llama-3.2-3B-Instruct, BSA (Budget Size of Attention)=1,0242026.02 | 79.36 | — | |
| QUOKABackbone=GPT-OSS-20B, BSA (Budget Size of Attention)=1,0242026.02 | 78.92 | — | |
| SampleAttnBackbone=Llama-3.2-3B-Instruct, BSA (Budget Size of Attention)=1,0242026.02 | 78.25 | — | |
| SparQBackbone=Qwen-2.5-3B-Instruct, BSA (Budget Size of Attention)=1,0242026.02 | 78.07 | — | |
| SampleAttnBackbone=Qwen-2.5-3B-Instruct, BSA (Budget Size of Attention)=1,0242026.02 | 77.17 | — | |
| SampleAttnBackbone=GPT-OSS-20B, BSA (Budget Size of Attention)=1,0242026.02 | 76.2 | — | |
| LokiBackbone=GPT-OSS-20B, BSA (Budget Size of Attention)=1,0242026.02 | 75.48 | — | |
| LessIsMoreBackbone=Llama-3.2-3B-Instruct, BSA (Budget Size of Attention)=1,0242026.02 | 75.15 | — | |
| LokiBackbone=Llama-3.2-3B-Instruct, BSA (Budget Size of Attention)=1,0242026.02 | 74.84 | — | |
| LokiBackbone=Qwen-2.5-3B-Instruct, BSA (Budget Size of Attention)=1,0242026.02 | 74.4 | — | |
| SparQBackbone=GPT-OSS-20B, BSA (Budget Size of Attention)=1,0242026.02 | 70.07 | — | |
| KeyDiffBackbone=GPT-OSS-20B, BSA (Budget Size of Attention)=1,0242026.02 | 69.58 | — | |
| LessIsMoreBackbone=GPT-OSS-20B, BSA (Budget Size of Attention)=1,0242026.02 | 67.35 | — | |
| LessIsMoreBackbone=Qwen-3-4B, BSA (Budget Size of Attention)=1,0242026.02 | 65.55 | — | |
| KeyDiffBackbone=Smollm3, BSA (Budget Size of Attention)=1,0242026.02 | 62.85 | — | |
| SnapKVBackbone=GPT-OSS-20B, BSA (Budget Size of Attention)=1,0242026.02 | 58.18 | — | |
| KeyDiffBackbone=Llama-3.2-3B-Instruct, BSA (Budget Size of Attention)=1,0242026.02 | 53.34 | — | |
| SnapKVBackbone=Smollm3, BSA (Budget Size of Attention)=1,0242026.02 | 43.72 | — | |
| KeyDiffBackbone=Qwen-3-4B, BSA (Budget Size of Attention)=1,0242026.02 | 37.29 | — | |
| LessIsMoreBackbone=Qwen-2.5-3B-Instruct, BSA (Budget Size of Attention)=1,0242026.02 | 36.66 | — | |
| KeyDiffBackbone=Qwen-2.5-3B-Instruct, BSA (Budget Size of Attention)=1,0242026.02 | 34.09 | — | |
| SnapKVBackbone=Llama-3.2-3B-Instruct, BSA (Budget Size of Attention)=1,0242026.02 | 29.15 | — | |
| SnapKVBackbone=Qwen-3-4B, BSA (Budget Size of Attention)=1,0242026.02 | 27.29 | — | |
| SnapKVBackbone=Qwen-2.5-3B-Instruct, BSA (Budget Size of Attention)=1,0242026.02 | 19.25 | — | |
| Full RepetitionBackbone=Qwen 2.5-3B2026.07 | — | 79.2 | |
| LLMLinguaBackbone=Qwen 2.5-3B2026.07 | — | 69.2 | |
| No RepetitionBackbone=Qwen 2.5-3B2026.07 | — | 77.7 | |
| PARTREPBackbone=Qwen 2.5-3B, tau=0.3, local-context token window expansion=true2026.07 | — | 80 |