Long-context Reasoning on BABILong 4k
38.5Accuracy (BABILong 4k)Full (exact)
Evaluation Results
| Method | Links | |
|---|---|---|
| Full (exact)Model=QWEN3-1.7B2026.04 | 38.5 | |
| Sink + Tail + Top-KModel=QWEN3-1.7B, Load Budget=5%2026.04 | 37 | |
| Sink + Tail + Top-KModel=QWEN3-1.7B, Load Budget=4%2026.04 | 36.7 | |
| Sink + Tail + Top-KModel=QWEN3-1.7B, Load Budget=3%2026.04 | 36 | |
| MAGICPIGModel=QWEN3-1.7B, Load Budget=3%2026.04 | 35.6 | |
| Sink + Tail + Top-K + ϕModel=QWEN3-1.7B, Load Budget=5%2026.04 | 34.3 | |
| Sink + Tail + Top-K + ϕModel=QWEN3-1.7B, Load Budget=4%2026.04 | 33.9 | |
| MAGICPIGModel=QWEN3-1.7B, Load Budget=5%2026.04 | 33.9 | |
| MAGICPIGModel=QWEN3-1.7B, Load Budget=4%2026.04 | 33.8 | |
| Sink + Tail + Top-K + ϕModel=QWEN3-1.7B, Load Budget=3%2026.04 | 33.7 | |
| SNAPKVModel=QWEN3-1.7B, Load Budget=4%2026.04 | 33.2 | |
| Full (exact)Model=LLAMA-3.2-1B-INSTRUCT2026.04 | 33.1 | |
| SNAPKVModel=QWEN3-1.7B, Load Budget=5%2026.04 | 32.6 | |
| Sink + Tail + Top-K + ϕModel=LLAMA-3.2-1B-INSTRUCT, Load Budget=4%2026.04 | 32.3 | |
| SNAPKVModel=LLAMA-3.2-1B-INSTRUCT, Load Budget=5%2026.04 | 32.3 | |
| Sink + Tail + Top-K + ϕModel=LLAMA-3.2-1B-INSTRUCT, Load Budget=2%2026.04 | 32.2 | |
| SNAPKVModel=QWEN3-1.7B, Load Budget=3%2026.04 | 32.1 | |
| Sink + Tail + Top-K + ϕModel=LLAMA-3.2-1B-INSTRUCT, Load Budget=3%2026.04 | 32 | |
| Sink + Tail + Top-K + ϕModel=LLAMA-3.2-1B-INSTRUCT, Load Budget=5%2026.04 | 32 | |
| Sink + Tail + Top-KModel=LLAMA-3.2-1B-INSTRUCT, Load Budget=5%2026.04 | 31.7 | |
| SNAPKVModel=LLAMA-3.2-1B-INSTRUCT, Load Budget=3%2026.04 | 31.6 | |
| MAGICPIGModel=LLAMA-3.2-1B-INSTRUCT, Load Budget=5%2026.04 | 31.5 | |
| SNAPKVModel=LLAMA-3.2-1B-INSTRUCT, Load Budget=4%2026.04 | 31.5 | |
| MAGICPIGModel=LLAMA-3.2-1B-INSTRUCT, Load Budget=4%2026.04 | 31.1 | |
| Sink + Tail + Top-KModel=LLAMA-3.2-1B-INSTRUCT, Load Budget=4%2026.04 | 31 | |
| MAGICPIGModel=LLAMA-3.2-1B-INSTRUCT, Load Budget=3%2026.04 | 31 | |
| Sink + Tail + Top-KModel=LLAMA-3.2-1B-INSTRUCT, Load Budget=3%2026.04 | 30.8 | |
| Sink + Tail + Top-KModel=LLAMA-3.2-1B-INSTRUCT, Load Budget=2%2026.04 | 30.6 | |
| MAGICPIGModel=LLAMA-3.2-1B-INSTRUCT, Load Budget=2%2026.04 | 29.7 | |
| PYRAMIDKVModel=QWEN3-1.7B, Load Budget=5%2026.04 | 29.7 | |
| PYRAMIDKVModel=QWEN3-1.7B, Load Budget=3%2026.04 | 29.2 | |
| SNAPKVModel=LLAMA-3.2-1B-INSTRUCT, Load Budget=2%2026.04 | 29.1 | |
| QUESTModel=LLAMA-3.2-1B-INSTRUCT, Load Budget=5%2026.04 | 28.7 | |
| PYRAMIDKVModel=QWEN3-1.7B, Load Budget=4%2026.04 | 28 | |
| PYRAMIDKVModel=LLAMA-3.2-1B-INSTRUCT, Load Budget=5%2026.04 | 27 | |
| QUESTModel=LLAMA-3.2-1B-INSTRUCT, Load Budget=4%2026.04 | 26.7 | |
| QUESTModel=LLAMA-3.2-1B-INSTRUCT, Load Budget=3%2026.04 | 25.8 | |
| PYRAMIDKVModel=LLAMA-3.2-1B-INSTRUCT, Load Budget=4%2026.04 | 25.8 | |
| QUESTModel=QWEN3-1.7B, Load Budget=5%2026.04 | 25.8 | |
| PYRAMIDKVModel=LLAMA-3.2-1B-INSTRUCT, Load Budget=3%2026.04 | 25.6 | |
| PYRAMIDKVModel=LLAMA-3.2-1B-INSTRUCT, Load Budget=2%2026.04 | 23 | |
| QUESTModel=LLAMA-3.2-1B-INSTRUCT, Load Budget=2%2026.04 | 22.6 | |
| QUESTModel=QWEN3-1.7B, Load Budget=4%2026.04 | 22.6 | |
| QUESTModel=QWEN3-1.7B, Load Budget=3%2026.04 | 20.8 | |
| STREAMINGLLMModel=LLAMA-3.2-1B-INSTRUCT, Load Budget=5%2026.04 | 16.5 | |
| STREAMINGLLMModel=LLAMA-3.2-1B-INSTRUCT, Load Budget=4%2026.04 | 15.6 | |
| STREAMINGLLMModel=LLAMA-3.2-1B-INSTRUCT, Load Budget=3%2026.04 | 15.1 | |
| STREAMINGLLMModel=LLAMA-3.2-1B-INSTRUCT, Load Budget=2%2026.04 | 13.6 | |
| STREAMINGLLMModel=QWEN3-1.7B, Load Budget=5%2026.04 | 12.7 | |
| STREAMINGLLMModel=QWEN3-1.7B, Load Budget=4%2026.04 | 11.4 | |
| STREAMINGLLMModel=QWEN3-1.7B, Load Budget=3%2026.04 | 10.8 |