Long-context evaluation on RULER 8k
91.07ScoreQUOKA
Evaluation results
| Method | Links | |
|---|---|---|
| QUOKABackbone=Qwen-3-4B, BSA (Budget Size of Attention)=1,0242026.02 | 91.07 | |
| QUOKABackbone=Llama-3.2-3B-Instruct, BSA (Budget Size of Attention)=1,0242026.02 | 80.19 | |
| QUOKABackbone=Smollm3, BSA (Budget Size of Attention)=1,0242026.02 | 79.94 | |
| QUOKABackbone=GPT-OSS-20B, BSA (Budget Size of Attention)=1,0242026.02 | 79.19 | |
| QUOKABackbone=Qwen-2.5-3B-Instruct, BSA (Budget Size of Attention)=1,0242026.02 | 74.27 | |
| SampleAttnBackbone=Qwen-3-4B, BSA (Budget Size of Attention)=1,0242026.02 | 72.46 | |
| SampleAttnBackbone=GPT-OSS-20B, BSA (Budget Size of Attention)=1,0242026.02 | 70.35 | |
| SparQBackbone=Qwen-3-4B, BSA (Budget Size of Attention)=1,0242026.02 | 68.97 | |
| SampleAttnBackbone=Smollm3, BSA (Budget Size of Attention)=1,0242026.02 | 66.44 | |
| LokiBackbone=GPT-OSS-20B, BSA (Budget Size of Attention)=1,0242026.02 | 65.36 | |
| LokiBackbone=Qwen-3-4B, BSA (Budget Size of Attention)=1,0242026.02 | 65.19 | |
| LokiBackbone=Smollm3, BSA (Budget Size of Attention)=1,0242026.02 | 64.2 | |
| SampleAttnBackbone=Llama-3.2-3B-Instruct, BSA (Budget Size of Attention)=1,0242026.02 | 61.14 | |
| SampleAttnBackbone=Qwen-2.5-3B-Instruct, BSA (Budget Size of Attention)=1,0242026.02 | 60.88 | |
| SparQBackbone=Llama-3.2-3B-Instruct, BSA (Budget Size of Attention)=1,0242026.02 | 60.8 | |
| LokiBackbone=Qwen-2.5-3B-Instruct, BSA (Budget Size of Attention)=1,0242026.02 | 60.09 | |
| SparQBackbone=Qwen-2.5-3B-Instruct, BSA (Budget Size of Attention)=1,0242026.02 | 59.87 | |
| SparQBackbone=Smollm3, BSA (Budget Size of Attention)=1,0242026.02 | 57.62 | |
| LokiBackbone=Llama-3.2-3B-Instruct, BSA (Budget Size of Attention)=1,0242026.02 | 56.5 | |
| LessIsMoreBackbone=GPT-OSS-20B, BSA (Budget Size of Attention)=1,0242026.02 | 54.49 | |
| SparQBackbone=GPT-OSS-20B, BSA (Budget Size of Attention)=1,0242026.02 | 54 | |
| KeyDiffBackbone=Smollm3, BSA (Budget Size of Attention)=1,0242026.02 | 51.6 | |
| LessIsMoreBackbone=Smollm3, BSA (Budget Size of Attention)=1,0242026.02 | 50.17 | |
| LessIsMoreBackbone=Llama-3.2-3B-Instruct, BSA (Budget Size of Attention)=1,0242026.02 | 49.23 | |
| LessIsMoreBackbone=Qwen-3-4B, BSA (Budget Size of Attention)=1,0242026.02 | 42.75 | |
| SnapKVBackbone=Smollm3, BSA (Budget Size of Attention)=1,0242026.02 | 35.98 | |
| SnapKVBackbone=GPT-OSS-20B, BSA (Budget Size of Attention)=1,0242026.02 | 31.45 | |
| KeyDiffBackbone=Llama-3.2-3B-Instruct, BSA (Budget Size of Attention)=1,0242026.02 | 31.1 | |
| KeyDiffBackbone=GPT-OSS-20B, BSA (Budget Size of Attention)=1,0242026.02 | 28.76 | |
| KeyDiffBackbone=Qwen-3-4B, BSA (Budget Size of Attention)=1,0242026.02 | 27.15 | |
| KeyDiffBackbone=Qwen-2.5-3B-Instruct, BSA (Budget Size of Attention)=1,0242026.02 | 20.68 | |
| LessIsMoreBackbone=Qwen-2.5-3B-Instruct, BSA (Budget Size of Attention)=1,0242026.02 | 20.21 | |
| SnapKVBackbone=Qwen-3-4B, BSA (Budget Size of Attention)=1,0242026.02 | 17.15 | |
| SnapKVBackbone=Llama-3.2-3B-Instruct, BSA (Budget Size of Attention)=1,0242026.02 | 13.7 | |
| SnapKVBackbone=Qwen-2.5-3B-Instruct, BSA (Budget Size of Attention)=1,0242026.02 | 12.4 |