Accuracy on Long-context Understanding on LongBench
103AccuracyQUOKA
Evaluation results
| Method | Links | |
|---|---|---|
| QUOKAModel=Smollm3, Budget (BSA)=10242026.02 | 103 | |
| QUOKAModel=Smollm3, Budget (BSA)=20482026.02 | 102.8 | |
| QUOKAModel=Smollm3, Budget (BSA)=5122026.02 | 99.8 | |
| QUOKAModel=Qwen3-4B, Budget (BSA)=20482026.02 | 99.5 | |
| QUOKAModel=Qwen3-4B, Budget (BSA)=10242026.02 | 99.2 | |
| QUOKAModel=Llama3.2-3B-Instruct, Budget (BSA)=20482026.02 | 98.6 | |
| QUOKAModel=Qwen2.5-3B-Instruct, Budget (BSA)=20482026.02 | 97.7 | |
| QUOKAModel=Llama3.2-3B-Instruct, Budget (BSA)=10242026.02 | 97.2 | |
| QUOKAModel=Qwen3-4B, Budget (BSA)=5122026.02 | 96.6 | |
| SampleAttnModel=Smollm3, Budget (BSA)=20482026.02 | 96.6 | |
| SampleAttnModel=Qwen3-4B, Budget (BSA)=20482026.02 | 94.7 | |
| QUOKAModel=Llama3.2-3B-Instruct, Budget (BSA)=5122026.02 | 94.5 | |
| QUOKAModel=Qwen2.5-3B-Instruct, Budget (BSA)=10242026.02 | 94.5 | |
| SampleAttnModel=Smollm3, Budget (BSA)=10242026.02 | 92.9 | |
| SparQModel=Smollm3, Budget (BSA)=20482026.02 | 92.2 | |
| LessIsMoreModel=Smollm3, Budget (BSA)=20482026.02 | 91.8 | |
| SampleAttnModel=Llama3.2-3B-Instruct, Budget (BSA)=20482026.02 | 90.1 | |
| SampleAttnModel=Qwen3-4B, Budget (BSA)=10242026.02 | 87.5 | |
| SparQModel=Qwen3-4B, Budget (BSA)=20482026.02 | 87.2 | |
| LokiModel=Qwen3-4B, Budget (BSA)=20482026.02 | 87.2 | |
| QUOKAModel=Qwen2.5-3B-Instruct, Budget (BSA)=5122026.02 | 86.9 | |
| LessIsMoreModel=Qwen3-4B, Budget (BSA)=20482026.02 | 86.8 | |
| SampleAttnModel=Smollm3, Budget (BSA)=5122026.02 | 85.6 | |
| LessIsMoreModel=Llama3.2-3B-Instruct, Budget (BSA)=20482026.02 | 85 | |
| SparQModel=Llama3.2-3B-Instruct, Budget (BSA)=20482026.02 | 84.2 | |
| LokiModel=Llama3.2-3B-Instruct, Budget (BSA)=20482026.02 | 84.2 | |
| LessIsMoreModel=Smollm3, Budget (BSA)=10242026.02 | 84.2 | |
| SampleAttnModel=Qwen2.5-3B-Instruct, Budget (BSA)=20482026.02 | 82.8 | |
| SparQModel=Qwen2.5-3B-Instruct, Budget (BSA)=20482026.02 | 80.8 | |
| SparQModel=Smollm3, Budget (BSA)=10242026.02 | 80.5 | |
| SparQModel=Llama3.2-3B-Instruct, Budget (BSA)=10242026.02 | 80.2 | |
| LokiModel=Smollm3, Budget (BSA)=10242026.02 | 80.1 | |
| SampleAttnModel=Llama3.2-3B-Instruct, Budget (BSA)=10242026.02 | 80 | |
| LessIsMoreModel=Llama3.2-3B-Instruct, Budget (BSA)=10242026.02 | 78.8 | |
| LokiModel=Qwen2.5-3B-Instruct, Budget (BSA)=20482026.02 | 78.7 | |
| SparQModel=Qwen3-4B, Budget (BSA)=10242026.02 | 78.2 | |
| LokiModel=Qwen3-4B, Budget (BSA)=10242026.02 | 78.2 | |
| LessIsMoreModel=Qwen3-4B, Budget (BSA)=10242026.02 | 77.3 | |
| LessIsMoreModel=Smollm3, Budget (BSA)=5122026.02 | 76.5 | |
| LokiModel=Llama3.2-3B-Instruct, Budget (BSA)=10242026.02 | 75.7 | |
| SampleAttnModel=Qwen2.5-3B-Instruct, Budget (BSA)=10242026.02 | 75.6 | |
| SampleAttnModel=Qwen3-4B, Budget (BSA)=5122026.02 | 75.5 | |
| SampleAttnModel=Llama3.2-3B-Instruct, Budget (BSA)=5122026.02 | 73.8 | |
| SparQModel=Qwen2.5-3B-Instruct, Budget (BSA)=10242026.02 | 72.6 | |
| SparQModel=Smollm3, Budget (BSA)=5122026.02 | 72.5 | |
| SparQModel=Llama3.2-3B-Instruct, Budget (BSA)=5122026.02 | 72.1 | |
| LessIsMoreModel=Llama3.2-3B-Instruct, Budget (BSA)=5122026.02 | 70.3 | |
| LokiModel=Llama3.2-3B-Instruct, Budget (BSA)=5122026.02 | 68.6 | |
| SparQModel=Qwen3-4B, Budget (BSA)=5122026.02 | 68.6 | |
| LokiModel=Qwen2.5-3B-Instruct, Budget (BSA)=10242026.02 | 67.1 | |
| LessIsMoreModel=Qwen3-4B, Budget (BSA)=5122026.02 | 66.5 | |
| SampleAttnModel=Qwen2.5-3B-Instruct, Budget (BSA)=5122026.02 | 66 | |
| LessIsMoreModel=Qwen2.5-3B-Instruct, Budget (BSA)=20482026.02 | 65.9 | |
| SparQModel=Qwen2.5-3B-Instruct, Budget (BSA)=5122026.02 | 63.6 | |
| LokiModel=Qwen3-4B, Budget (BSA)=5122026.02 | 62.2 | |
| LokiModel=Smollm3, Budget (BSA)=20482026.02 | 62.2 | |
| LokiModel=Qwen2.5-3B-Instruct, Budget (BSA)=5122026.02 | 58.9 | |
| LessIsMoreModel=Qwen2.5-3B-Instruct, Budget (BSA)=10242026.02 | 55.6 | |
| LessIsMoreModel=Qwen2.5-3B-Instruct, Budget (BSA)=5122026.02 | 46.1 | |
| LokiModel=Smollm3, Budget (BSA)=5122026.02 | 38.4 |