Long-context Reasoning on BABILong 8k
34.7AccuracyFull (exact)
Evaluation Results
| Method | Links | |
|---|---|---|
| Full (exact)Model=QWEN3-1.7B2026.04 | 34.7 | |
| Sink + Tail + Top-K + ϕModel=QWEN3-1.7B, Load Budget=5%2026.04 | 33 | |
| MAGICPIGModel=QWEN3-1.7B, Load Budget=5%2026.04 | 33 | |
| Sink + Tail + Top-K + ϕModel=QWEN3-1.7B, Load Budget=4%2026.04 | 32.5 | |
| MAGICPIGModel=QWEN3-1.7B, Load Budget=3%2026.04 | 32.5 | |
| Sink + Tail + Top-K + ϕModel=QWEN3-1.7B, Load Budget=3%2026.04 | 32.3 | |
| Sink + Tail + Top-KModel=QWEN3-1.7B, Load Budget=5%2026.04 | 32.2 | |
| Sink + Tail + Top-KModel=QWEN3-1.7B, Load Budget=4%2026.04 | 32.1 | |
| MAGICPIGModel=QWEN3-1.7B, Load Budget=4%2026.04 | 31.9 | |
| Sink + Tail + Top-KModel=QWEN3-1.7B, Load Budget=3%2026.04 | 31.6 | |
| Sink + Tail + Top-KModel=QWEN3-1.7B, Load Budget=2%2026.04 | 31.3 | |
| Sink + Tail + Top-K + ϕModel=QWEN3-1.7B, Load Budget=2%2026.04 | 31.3 | |
| SNAPKVModel=QWEN3-1.7B, Load Budget=5%2026.04 | 30.6 | |
| SNAPKVModel=QWEN3-1.7B, Load Budget=3%2026.04 | 30.3 | |
| MAGICPIGModel=QWEN3-1.7B, Load Budget=2%2026.04 | 30.1 | |
| SNAPKVModel=QWEN3-1.7B, Load Budget=4%2026.04 | 29.9 | |
| SNAPKVModel=QWEN3-1.7B, Load Budget=2%2026.04 | 29.5 | |
| Full (exact)Model=LLAMA-3.2-1B-INSTRUCT2026.04 | 28.9 | |
| PYRAMIDKVModel=QWEN3-1.7B, Load Budget=5%2026.04 | 28.9 | |
| SNAPKVModel=LLAMA-3.2-1B-INSTRUCT, Load Budget=5%2026.04 | 28.8 | |
| MAGICPIGModel=LLAMA-3.2-1B-INSTRUCT, Load Budget=3%2026.04 | 28.5 | |
| SNAPKVModel=LLAMA-3.2-1B-INSTRUCT, Load Budget=4%2026.04 | 28.4 | |
| SNAPKVModel=LLAMA-3.2-1B-INSTRUCT, Load Budget=3%2026.04 | 28.1 | |
| MAGICPIGModel=LLAMA-3.2-1B-INSTRUCT, Load Budget=5%2026.04 | 27.9 | |
| PYRAMIDKVModel=QWEN3-1.7B, Load Budget=4%2026.04 | 27.9 | |
| Sink + Tail + Top-K + ϕModel=LLAMA-3.2-1B-INSTRUCT, Load Budget=1%2026.04 | 27.7 | |
| Sink + Tail + Top-K + ϕModel=LLAMA-3.2-1B-INSTRUCT, Load Budget=5%2026.04 | 27.5 | |
| Sink + Tail + Top-K + ϕModel=LLAMA-3.2-1B-INSTRUCT, Load Budget=4%2026.04 | 27.3 | |
| MAGICPIGModel=LLAMA-3.2-1B-INSTRUCT, Load Budget=4%2026.04 | 27.3 | |
| Sink + Tail + Top-K + ϕModel=LLAMA-3.2-1B-INSTRUCT, Load Budget=3%2026.04 | 26.9 | |
| Sink + Tail + Top-K + ϕModel=LLAMA-3.2-1B-INSTRUCT, Load Budget=2%2026.04 | 26.8 | |
| SNAPKVModel=LLAMA-3.2-1B-INSTRUCT, Load Budget=2%2026.04 | 26.8 | |
| Sink + Tail + Top-KModel=LLAMA-3.2-1B-INSTRUCT, Load Budget=5%2026.04 | 26.7 | |
| MAGICPIGModel=LLAMA-3.2-1B-INSTRUCT, Load Budget=2%2026.04 | 26.7 | |
| Sink + Tail + Top-KModel=LLAMA-3.2-1B-INSTRUCT, Load Budget=3%2026.04 | 26.5 | |
| Sink + Tail + Top-KModel=LLAMA-3.2-1B-INSTRUCT, Load Budget=4%2026.04 | 26.5 | |
| PYRAMIDKVModel=QWEN3-1.7B, Load Budget=3%2026.04 | 26.5 | |
| QUESTModel=LLAMA-3.2-1B-INSTRUCT, Load Budget=4%2026.04 | 26.4 | |
| QUESTModel=QWEN3-1.7B, Load Budget=5%2026.04 | 26.3 | |
| Sink + Tail + Top-KModel=LLAMA-3.2-1B-INSTRUCT, Load Budget=2%2026.04 | 25.7 | |
| Sink + Tail + Top-KModel=LLAMA-3.2-1B-INSTRUCT, Load Budget=1%2026.04 | 25.4 | |
| PYRAMIDKVModel=LLAMA-3.2-1B-INSTRUCT, Load Budget=5%2026.04 | 25.4 | |
| QUESTModel=LLAMA-3.2-1B-INSTRUCT, Load Budget=5%2026.04 | 25.2 | |
| SNAPKVModel=LLAMA-3.2-1B-INSTRUCT, Load Budget=1%2026.04 | 25.2 | |
| QUESTModel=QWEN3-1.7B, Load Budget=4%2026.04 | 25.2 | |
| PYRAMIDKVModel=LLAMA-3.2-1B-INSTRUCT, Load Budget=4%2026.04 | 24.6 | |
| QUESTModel=LLAMA-3.2-1B-INSTRUCT, Load Budget=3%2026.04 | 24.5 | |
| MAGICPIGModel=LLAMA-3.2-1B-INSTRUCT, Load Budget=1%2026.04 | 24.5 | |
| PYRAMIDKVModel=QWEN3-1.7B, Load Budget=2%2026.04 | 24.4 | |
| PYRAMIDKVModel=LLAMA-3.2-1B-INSTRUCT, Load Budget=3%2026.04 | 24.3 | |
| QUESTModel=LLAMA-3.2-1B-INSTRUCT, Load Budget=2%2026.04 | 22.8 | |
| PYRAMIDKVModel=LLAMA-3.2-1B-INSTRUCT, Load Budget=2%2026.04 | 22.8 | |
| QUESTModel=QWEN3-1.7B, Load Budget=3%2026.04 | 22.2 | |
| PYRAMIDKVModel=LLAMA-3.2-1B-INSTRUCT, Load Budget=1%2026.04 | 20.2 | |
| QUESTModel=QWEN3-1.7B, Load Budget=2%2026.04 | 19.4 | |
| QUESTModel=LLAMA-3.2-1B-INSTRUCT, Load Budget=1%2026.04 | 18.7 | |
| STREAMINGLLMModel=LLAMA-3.2-1B-INSTRUCT, Load Budget=5%2026.04 | 15 | |
| STREAMINGLLMModel=LLAMA-3.2-1B-INSTRUCT, Load Budget=4%2026.04 | 14.3 | |
| STREAMINGLLMModel=LLAMA-3.2-1B-INSTRUCT, Load Budget=3%2026.04 | 13.2 | |
| STREAMINGLLMModel=QWEN3-1.7B, Load Budget=5%2026.04 | 12.6 | |
| STREAMINGLLMModel=LLAMA-3.2-1B-INSTRUCT, Load Budget=2%2026.04 | 11.9 | |
| STREAMINGLLMModel=QWEN3-1.7B, Load Budget=4%2026.04 | 11.4 | |
| STREAMINGLLMModel=LLAMA-3.2-1B-INSTRUCT, Load Budget=1%2026.04 | 11.3 | |
| STREAMINGLLMModel=QWEN3-1.7B, Load Budget=3%2026.04 | 9.8 | |
| STREAMINGLLMModel=QWEN3-1.7B, Load Budget=2%2026.04 | 8.3 |