List your company
ResearchFormerly WizWand

Long-context Reasoning on BABILong 4k

38.5Accuracy (BABILong 4k)Full (exact)
9.69217.17124.6532.129Apr 7, 2026
Updated 6mo ago

Evaluation results

MethodLinks
Full (exact)Model=QWEN3-1.7B2026.0438.5
Sink + Tail + Top-KModel=QWEN3-1.7B, Load Budget=5%2026.0437
Sink + Tail + Top-KModel=QWEN3-1.7B, Load Budget=4%2026.0436.7
Sink + Tail + Top-KModel=QWEN3-1.7B, Load Budget=3%2026.0436
MAGICPIGModel=QWEN3-1.7B, Load Budget=3%2026.0435.6
Sink + Tail + Top-K + ϕModel=QWEN3-1.7B, Load Budget=5%2026.0434.3
Sink + Tail + Top-K + ϕModel=QWEN3-1.7B, Load Budget=4%2026.0433.9
MAGICPIGModel=QWEN3-1.7B, Load Budget=5%2026.0433.9
MAGICPIGModel=QWEN3-1.7B, Load Budget=4%2026.0433.8
Sink + Tail + Top-K + ϕModel=QWEN3-1.7B, Load Budget=3%2026.0433.7
SNAPKVModel=QWEN3-1.7B, Load Budget=4%2026.0433.2
Full (exact)Model=LLAMA-3.2-1B-INSTRUCT2026.0433.1
SNAPKVModel=QWEN3-1.7B, Load Budget=5%2026.0432.6
Sink + Tail + Top-K + ϕModel=LLAMA-3.2-1B-INSTRUCT, Load Budget=4%2026.0432.3
SNAPKVModel=LLAMA-3.2-1B-INSTRUCT, Load Budget=5%2026.0432.3
Sink + Tail + Top-K + ϕModel=LLAMA-3.2-1B-INSTRUCT, Load Budget=2%2026.0432.2
SNAPKVModel=QWEN3-1.7B, Load Budget=3%2026.0432.1
Sink + Tail + Top-K + ϕModel=LLAMA-3.2-1B-INSTRUCT, Load Budget=3%2026.0432
Sink + Tail + Top-K + ϕModel=LLAMA-3.2-1B-INSTRUCT, Load Budget=5%2026.0432
Sink + Tail + Top-KModel=LLAMA-3.2-1B-INSTRUCT, Load Budget=5%2026.0431.7
SNAPKVModel=LLAMA-3.2-1B-INSTRUCT, Load Budget=3%2026.0431.6
MAGICPIGModel=LLAMA-3.2-1B-INSTRUCT, Load Budget=5%2026.0431.5
SNAPKVModel=LLAMA-3.2-1B-INSTRUCT, Load Budget=4%2026.0431.5
MAGICPIGModel=LLAMA-3.2-1B-INSTRUCT, Load Budget=4%2026.0431.1
Sink + Tail + Top-KModel=LLAMA-3.2-1B-INSTRUCT, Load Budget=4%2026.0431
MAGICPIGModel=LLAMA-3.2-1B-INSTRUCT, Load Budget=3%2026.0431
Sink + Tail + Top-KModel=LLAMA-3.2-1B-INSTRUCT, Load Budget=3%2026.0430.8
Sink + Tail + Top-KModel=LLAMA-3.2-1B-INSTRUCT, Load Budget=2%2026.0430.6
MAGICPIGModel=LLAMA-3.2-1B-INSTRUCT, Load Budget=2%2026.0429.7
PYRAMIDKVModel=QWEN3-1.7B, Load Budget=5%2026.0429.7
PYRAMIDKVModel=QWEN3-1.7B, Load Budget=3%2026.0429.2
SNAPKVModel=LLAMA-3.2-1B-INSTRUCT, Load Budget=2%2026.0429.1
QUESTModel=LLAMA-3.2-1B-INSTRUCT, Load Budget=5%2026.0428.7
PYRAMIDKVModel=QWEN3-1.7B, Load Budget=4%2026.0428
PYRAMIDKVModel=LLAMA-3.2-1B-INSTRUCT, Load Budget=5%2026.0427
QUESTModel=LLAMA-3.2-1B-INSTRUCT, Load Budget=4%2026.0426.7
QUESTModel=LLAMA-3.2-1B-INSTRUCT, Load Budget=3%2026.0425.8
PYRAMIDKVModel=LLAMA-3.2-1B-INSTRUCT, Load Budget=4%2026.0425.8
QUESTModel=QWEN3-1.7B, Load Budget=5%2026.0425.8
PYRAMIDKVModel=LLAMA-3.2-1B-INSTRUCT, Load Budget=3%2026.0425.6
PYRAMIDKVModel=LLAMA-3.2-1B-INSTRUCT, Load Budget=2%2026.0423
QUESTModel=LLAMA-3.2-1B-INSTRUCT, Load Budget=2%2026.0422.6
QUESTModel=QWEN3-1.7B, Load Budget=4%2026.0422.6
QUESTModel=QWEN3-1.7B, Load Budget=3%2026.0420.8
STREAMINGLLMModel=LLAMA-3.2-1B-INSTRUCT, Load Budget=5%2026.0416.5
STREAMINGLLMModel=LLAMA-3.2-1B-INSTRUCT, Load Budget=4%2026.0415.6
STREAMINGLLMModel=LLAMA-3.2-1B-INSTRUCT, Load Budget=3%2026.0415.1
STREAMINGLLMModel=LLAMA-3.2-1B-INSTRUCT, Load Budget=2%2026.0413.6
STREAMINGLLMModel=QWEN3-1.7B, Load Budget=5%2026.0412.7
STREAMINGLLMModel=QWEN3-1.7B, Load Budget=4%2026.0411.4
STREAMINGLLMModel=QWEN3-1.7B, Load Budget=3%2026.0410.8