Long-context language modeling on RULER 16K context
83Accuracy (RULER 16K)SINK + TAIL + TOP-K + ϕ
Evaluation results
| Method | Links | |
|---|---|---|
| SINK + TAIL + TOP-K + ϕModel Backbone=QWEN3-1.7B, Load Budget=2%2026.04 | 83 | |
| SINK + TAIL + TOP-KModel Backbone=QWEN3-1.7B, Load Budget=2%2026.04 | 82.9 | |
| SINK + TAIL + TOP-KModel Backbone=QWEN3-1.7B, Load Budget=3%2026.04 | 82.8 | |
| SINK + TAIL + TOP-KModel Backbone=QWEN3-1.7B, Load Budget=4%2026.04 | 82.8 | |
| SINK + TAIL + TOP-KModel Backbone=QWEN3-1.7B, Load Budget=5%2026.04 | 82.8 | |
| SINK + TAIL + TOP-K + ϕModel Backbone=QWEN3-1.7B, Load Budget=3%2026.04 | 82.7 | |
| Full (exact)Model Backbone=QWEN3-1.7B, Load Budget=100% (Full)2026.04 | 82.5 | |
| SINK + TAIL + TOP-K + ϕModel Backbone=QWEN3-1.7B, Load Budget=4%2026.04 | 82.5 | |
| SINK + TAIL + TOP-KModel Backbone=QWEN3-1.7B, Load Budget=1%2026.04 | 82.4 | |
| SINK + TAIL + TOP-K + ϕModel Backbone=QWEN3-1.7B, Load Budget=5%2026.04 | 82.4 | |
| SINK + TAIL + TOP-K + ϕModel Backbone=QWEN3-1.7B, Load Budget=1%2026.04 | 81.8 | |
| MAGICPIGModel Backbone=QWEN3-1.7B, Load Budget=3%2026.04 | 77.5 | |
| MAGICPIGModel Backbone=QWEN3-1.7B, Load Budget=4%2026.04 | 77.2 | |
| MAGICPIGModel Backbone=QWEN3-1.7B, Load Budget=5%2026.04 | 75.5 | |
| MAGICPIGModel Backbone=QWEN3-1.7B, Load Budget=2%2026.04 | 72.1 | |
| Full (exact)Model Backbone=LLAMA-3.2-1B-INSTRUCT, Load Budget=100% (Full)2026.04 | 66.5 | |
| SINK + TAIL + TOP-K + ϕModel Backbone=LLAMA-3.2-1B-INSTRUCT, Load Budget=5%2026.04 | 65.5 | |
| SINK + TAIL + TOP-K + ϕModel Backbone=LLAMA-3.2-1B-INSTRUCT, Load Budget=4%2026.04 | 65.3 | |
| SINK + TAIL + TOP-K + ϕModel Backbone=LLAMA-3.2-1B-INSTRUCT, Load Budget=3%2026.04 | 64.7 | |
| SINK + TAIL + TOP-KModel Backbone=LLAMA-3.2-1B-INSTRUCT, Load Budget=5%2026.04 | 64.6 | |
| SINK + TAIL + TOP-KModel Backbone=LLAMA-3.2-1B-INSTRUCT, Load Budget=4%2026.04 | 64.3 | |
| SINK + TAIL + TOP-K + ϕModel Backbone=LLAMA-3.2-1B-INSTRUCT, Load Budget=2%2026.04 | 64.1 | |
| MAGICPIGModel Backbone=QWEN3-1.7B, Load Budget=1%2026.04 | 63.7 | |
| SINK + TAIL + TOP-KModel Backbone=LLAMA-3.2-1B-INSTRUCT, Load Budget=3%2026.04 | 63.6 | |
| SINK + TAIL + TOP-K + ϕModel Backbone=LLAMA-3.2-1B-INSTRUCT, Load Budget=1%2026.04 | 63 | |
| SINK + TAIL + TOP-KModel Backbone=LLAMA-3.2-1B-INSTRUCT, Load Budget=2%2026.04 | 62.7 | |
| SINK + TAIL + TOP-KModel Backbone=LLAMA-3.2-1B-INSTRUCT, Load Budget=1%2026.04 | 60.6 | |
| QUESTModel Backbone=QWEN3-1.7B, Load Budget=5%2026.04 | 59.2 | |
| MAGICPIGModel Backbone=LLAMA-3.2-1B-INSTRUCT, Load Budget=3%2026.04 | 58.3 | |
| MAGICPIGModel Backbone=LLAMA-3.2-1B-INSTRUCT, Load Budget=4%2026.04 | 55.5 | |
| QUESTModel Backbone=QWEN3-1.7B, Load Budget=4%2026.04 | 54.5 | |
| MAGICPIGModel Backbone=LLAMA-3.2-1B-INSTRUCT, Load Budget=5%2026.04 | 53.6 | |
| QUESTModel Backbone=QWEN3-1.7B, Load Budget=3%2026.04 | 47.5 | |
| MAGICPIGModel Backbone=LLAMA-3.2-1B-INSTRUCT, Load Budget=2%2026.04 | 46.8 | |
| SNAPKVModel Backbone=QWEN3-1.7B, Load Budget=5%2026.04 | 46.6 | |
| SNAPKVModel Backbone=QWEN3-1.7B, Load Budget=4%2026.04 | 44 | |
| QUESTModel Backbone=LLAMA-3.2-1B-INSTRUCT, Load Budget=5%2026.04 | 43.3 | |
| PYRAMIDKVModel Backbone=QWEN3-1.7B, Load Budget=5%2026.04 | 42.9 | |
| MAGICPIGModel Backbone=LLAMA-3.2-1B-INSTRUCT, Load Budget=1%2026.04 | 41.2 | |
| SNAPKVModel Backbone=QWEN3-1.7B, Load Budget=3%2026.04 | 40.8 | |
| QUESTModel Backbone=LLAMA-3.2-1B-INSTRUCT, Load Budget=4%2026.04 | 40.6 | |
| QUESTModel Backbone=QWEN3-1.7B, Load Budget=2%2026.04 | 40.4 | |
| PYRAMIDKVModel Backbone=QWEN3-1.7B, Load Budget=4%2026.04 | 39.9 | |
| QUESTModel Backbone=LLAMA-3.2-1B-INSTRUCT, Load Budget=3%2026.04 | 37.9 | |
| PYRAMIDKVModel Backbone=QWEN3-1.7B, Load Budget=3%2026.04 | 37 | |
| SNAPKVModel Backbone=LLAMA-3.2-1B-INSTRUCT, Load Budget=5%2026.04 | 36.8 | |
| SNAPKVModel Backbone=QWEN3-1.7B, Load Budget=2%2026.04 | 36.6 | |
| SNAPKVModel Backbone=LLAMA-3.2-1B-INSTRUCT, Load Budget=4%2026.04 | 35.5 | |
| SNAPKVModel Backbone=LLAMA-3.2-1B-INSTRUCT, Load Budget=3%2026.04 | 33.8 | |
| QUESTModel Backbone=LLAMA-3.2-1B-INSTRUCT, Load Budget=2%2026.04 | 33.5 | |
| PYRAMIDKVModel Backbone=LLAMA-3.2-1B-INSTRUCT, Load Budget=5%2026.04 | 33.4 | |
| PYRAMIDKVModel Backbone=QWEN3-1.7B, Load Budget=2%2026.04 | 31.1 | |
| PYRAMIDKVModel Backbone=LLAMA-3.2-1B-INSTRUCT, Load Budget=4%2026.04 | 31 | |
| SNAPKVModel Backbone=LLAMA-3.2-1B-INSTRUCT, Load Budget=2%2026.04 | 30.5 | |
| QUESTModel Backbone=QWEN3-1.7B, Load Budget=1%2026.04 | 29.9 | |
| PYRAMIDKVModel Backbone=LLAMA-3.2-1B-INSTRUCT, Load Budget=3%2026.04 | 29.4 | |
| SNAPKVModel Backbone=QWEN3-1.7B, Load Budget=1%2026.04 | 26 | |
| QUESTModel Backbone=LLAMA-3.2-1B-INSTRUCT, Load Budget=1%2026.04 | 24.3 | |
| PYRAMIDKVModel Backbone=LLAMA-3.2-1B-INSTRUCT, Load Budget=2%2026.04 | 24 | |
| SNAPKVModel Backbone=LLAMA-3.2-1B-INSTRUCT, Load Budget=1%2026.04 | 22.1 | |
| PYRAMIDKVModel Backbone=QWEN3-1.7B, Load Budget=1%2026.04 | 16.7 | |
| PYRAMIDKVModel Backbone=LLAMA-3.2-1B-INSTRUCT, Load Budget=1%2026.04 | 9.1 | |
| STREAMINGLLMModel Backbone=QWEN3-1.7B, Load Budget=5%2026.04 | 7.9 | |
| STREAMINGLLMModel Backbone=LLAMA-3.2-1B-INSTRUCT, Load Budget=5%2026.04 | 6.8 | |
| STREAMINGLLMModel Backbone=LLAMA-3.2-1B-INSTRUCT, Load Budget=4%2026.04 | 6.5 | |
| STREAMINGLLMModel Backbone=QWEN3-1.7B, Load Budget=4%2026.04 | 6.4 | |
| STREAMINGLLMModel Backbone=LLAMA-3.2-1B-INSTRUCT, Load Budget=3%2026.04 | 5.3 | |
| STREAMINGLLMModel Backbone=LLAMA-3.2-1B-INSTRUCT, Load Budget=2%2026.04 | 4.9 | |
| STREAMINGLLMModel Backbone=QWEN3-1.7B, Load Budget=3%2026.04 | 4.4 | |
| STREAMINGLLMModel Backbone=LLAMA-3.2-1B-INSTRUCT, Load Budget=1%2026.04 | 4.2 | |
| STREAMINGLLMModel Backbone=QWEN3-1.7B, Load Budget=2%2026.04 | 3.6 | |
| STREAMINGLLMModel Backbone=QWEN3-1.7B, Load Budget=1%2026.04 | 3.1 |