Long-context Reasoning on BABILong 16k
29.8AccuracyFull (exact)
Evaluation Results
| Method | Links | |
|---|---|---|
| Full (exact)Model=QWEN3-1.7B2026.04 | 29.8 | |
| Sink + Tail + Top-KModel=QWEN3-1.7B, Load Budget=5%2026.04 | 28.8 | |
| MAGICPIGModel=QWEN3-1.7B, Load Budget=4%2026.04 | 28.4 | |
| Sink + Tail + Top-K + ϕModel=QWEN3-1.7B, Load Budget=5%2026.04 | 28.3 | |
| MAGICPIGModel=QWEN3-1.7B, Load Budget=5%2026.04 | 28.3 | |
| Sink + Tail + Top-KModel=QWEN3-1.7B, Load Budget=4%2026.04 | 28 | |
| MAGICPIGModel=QWEN3-1.7B, Load Budget=3%2026.04 | 27.8 | |
| Sink + Tail + Top-K + ϕModel=QWEN3-1.7B, Load Budget=4%2026.04 | 27.7 | |
| SNAPKVModel=QWEN3-1.7B, Load Budget=5%2026.04 | 27.5 | |
| Sink + Tail + Top-KModel=QWEN3-1.7B, Load Budget=3%2026.04 | 27.4 | |
| Sink + Tail + Top-K + ϕModel=QWEN3-1.7B, Load Budget=3%2026.04 | 27.3 | |
| SNAPKVModel=QWEN3-1.7B, Load Budget=4%2026.04 | 27.1 | |
| MAGICPIGModel=QWEN3-1.7B, Load Budget=2%2026.04 | 27 | |
| SNAPKVModel=QWEN3-1.7B, Load Budget=3%2026.04 | 26.9 | |
| Sink + Tail + Top-KModel=QWEN3-1.7B, Load Budget=2%2026.04 | 26.7 | |
| Sink + Tail + Top-K + ϕModel=QWEN3-1.7B, Load Budget=2%2026.04 | 26.5 | |
| SNAPKVModel=QWEN3-1.7B, Load Budget=2%2026.04 | 26.1 | |
| Sink + Tail + Top-KModel=QWEN3-1.7B, Load Budget=1%2026.04 | 25.9 | |
| PYRAMIDKVModel=QWEN3-1.7B, Load Budget=5%2026.04 | 25.4 | |
| MAGICPIGModel=QWEN3-1.7B, Load Budget=1%2026.04 | 24.9 | |
| Sink + Tail + Top-K + ϕModel=QWEN3-1.7B, Load Budget=1%2026.04 | 24.8 | |
| SNAPKVModel=QWEN3-1.7B, Load Budget=1%2026.04 | 24.6 | |
| Full (exact)Model=LLAMA-3.2-1B-INSTRUCT2026.04 | 24.4 | |
| PYRAMIDKVModel=QWEN3-1.7B, Load Budget=4%2026.04 | 23.9 | |
| SNAPKVModel=LLAMA-3.2-1B-INSTRUCT, Load Budget=2%2026.04 | 23.8 | |
| SNAPKVModel=LLAMA-3.2-1B-INSTRUCT, Load Budget=4%2026.04 | 23.8 | |
| PYRAMIDKVModel=QWEN3-1.7B, Load Budget=3%2026.04 | 23.8 | |
| SNAPKVModel=LLAMA-3.2-1B-INSTRUCT, Load Budget=3%2026.04 | 23.7 | |
| Sink + Tail + Top-K + ϕModel=LLAMA-3.2-1B-INSTRUCT, Load Budget=1%2026.04 | 23.6 | |
| MAGICPIGModel=LLAMA-3.2-1B-INSTRUCT, Load Budget=3%2026.04 | 23.6 | |
| SNAPKVModel=LLAMA-3.2-1B-INSTRUCT, Load Budget=5%2026.04 | 23.5 | |
| QUESTModel=QWEN3-1.7B, Load Budget=5%2026.04 | 23.5 | |
| MAGICPIGModel=LLAMA-3.2-1B-INSTRUCT, Load Budget=4%2026.04 | 23.4 | |
| Sink + Tail + Top-K + ϕModel=LLAMA-3.2-1B-INSTRUCT, Load Budget=2%2026.04 | 23 | |
| QUESTModel=LLAMA-3.2-1B-INSTRUCT, Load Budget=5%2026.04 | 23 | |
| MAGICPIGModel=LLAMA-3.2-1B-INSTRUCT, Load Budget=5%2026.04 | 23 | |
| Sink + Tail + Top-K + ϕModel=LLAMA-3.2-1B-INSTRUCT, Load Budget=3%2026.04 | 22.9 | |
| Sink + Tail + Top-K + ϕModel=LLAMA-3.2-1B-INSTRUCT, Load Budget=4%2026.04 | 22.9 | |
| Sink + Tail + Top-K + ϕModel=LLAMA-3.2-1B-INSTRUCT, Load Budget=5%2026.04 | 22.8 | |
| SNAPKVModel=LLAMA-3.2-1B-INSTRUCT, Load Budget=1%2026.04 | 22.8 | |
| MAGICPIGModel=LLAMA-3.2-1B-INSTRUCT, Load Budget=2%2026.04 | 22.7 | |
| QUESTModel=LLAMA-3.2-1B-INSTRUCT, Load Budget=4%2026.04 | 22.4 | |
| PYRAMIDKVModel=QWEN3-1.7B, Load Budget=2%2026.04 | 22.2 | |
| Sink + Tail + Top-KModel=LLAMA-3.2-1B-INSTRUCT, Load Budget=5%2026.04 | 22.1 | |
| QUESTModel=LLAMA-3.2-1B-INSTRUCT, Load Budget=3%2026.04 | 22 | |
| PYRAMIDKVModel=LLAMA-3.2-1B-INSTRUCT, Load Budget=4%2026.04 | 22 | |
| PYRAMIDKVModel=LLAMA-3.2-1B-INSTRUCT, Load Budget=5%2026.04 | 21.9 | |
| Sink + Tail + Top-KModel=LLAMA-3.2-1B-INSTRUCT, Load Budget=4%2026.04 | 21.8 | |
| QUESTModel=QWEN3-1.7B, Load Budget=4%2026.04 | 21.7 | |
| PYRAMIDKVModel=LLAMA-3.2-1B-INSTRUCT, Load Budget=3%2026.04 | 21.6 | |
| Sink + Tail + Top-KModel=LLAMA-3.2-1B-INSTRUCT, Load Budget=3%2026.04 | 21.3 | |
| MAGICPIGModel=LLAMA-3.2-1B-INSTRUCT, Load Budget=1%2026.04 | 21.3 | |
| PYRAMIDKVModel=LLAMA-3.2-1B-INSTRUCT, Load Budget=2%2026.04 | 21.3 | |
| PYRAMIDKVModel=QWEN3-1.7B, Load Budget=1%2026.04 | 21.1 | |
| Sink + Tail + Top-KModel=LLAMA-3.2-1B-INSTRUCT, Load Budget=2%2026.04 | 20.6 | |
| Sink + Tail + Top-KModel=LLAMA-3.2-1B-INSTRUCT, Load Budget=1%2026.04 | 20.4 | |
| QUESTModel=LLAMA-3.2-1B-INSTRUCT, Load Budget=2%2026.04 | 20.2 | |
| QUESTModel=QWEN3-1.7B, Load Budget=3%2026.04 | 20.2 | |
| PYRAMIDKVModel=LLAMA-3.2-1B-INSTRUCT, Load Budget=1%2026.04 | 19.5 | |
| QUESTModel=LLAMA-3.2-1B-INSTRUCT, Load Budget=1%2026.04 | 17.7 | |
| QUESTModel=QWEN3-1.7B, Load Budget=2%2026.04 | 17.4 | |
| QUESTModel=QWEN3-1.7B, Load Budget=1%2026.04 | 14.1 | |
| STREAMINGLLMModel=LLAMA-3.2-1B-INSTRUCT, Load Budget=5%2026.04 | 13.7 | |
| STREAMINGLLMModel=LLAMA-3.2-1B-INSTRUCT, Load Budget=4%2026.04 | 12.9 | |
| STREAMINGLLMModel=QWEN3-1.7B, Load Budget=5%2026.04 | 12.5 | |
| STREAMINGLLMModel=LLAMA-3.2-1B-INSTRUCT, Load Budget=3%2026.04 | 12.4 | |
| STREAMINGLLMModel=LLAMA-3.2-1B-INSTRUCT, Load Budget=2%2026.04 | 11.4 | |
| STREAMINGLLMModel=QWEN3-1.7B, Load Budget=4%2026.04 | 11.2 | |
| STREAMINGLLMModel=LLAMA-3.2-1B-INSTRUCT, Load Budget=1%2026.04 | 11 | |
| STREAMINGLLMModel=QWEN3-1.7B, Load Budget=3%2026.04 | 10.2 | |
| STREAMINGLLMModel=QWEN3-1.7B, Load Budget=2%2026.04 | 8.7 | |
| STREAMINGLLMModel=QWEN3-1.7B, Load Budget=1%2026.04 | 7.1 |