Long-context language modeling evaluation on RULER (8K Context)
89.59Average Accuracy (RULER 8K)Full
Evaluation Results
| Method | Links | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| FullContext Length=8K2026.03 | 89.59 | 100 | 100 | 99.4 | 99.4 | 99.25 | 83.4 | 96.64 | 69.83 | 58.4 | |
| MixedDimKV-HContext Length=8K2026.03 | 89.47 | 100 | 99.6 | 98.4 | 97.4 | 96.5 | 89.7 | 97.28 | 68.78 | 57.6 | |
| HeadKVContext Length=8K2026.03 | 89.08 | 100 | 99.2 | 99.2 | 99 | 98.85 | 94.75 | 89.92 | 65.6 | 55.2 | |
| MixedDimKVContext Length=8K2026.03 | 87.18 | 99.6 | 99.4 | 98.8 | 95.6 | 88.4 | 80.25 | 97.52 | 67.62 | 57.4 | |
| Full (exact)Model Backbone=QWEN3-1.7B, Load Budget=100% (Full)2026.04 | 85.5 | — | — | — | — | — | — | — | — | — | |
| SINK + TAIL + TOP-KModel Backbone=QWEN3-1.7B, Load Budget=4%2026.04 | 85.3 | — | — | — | — | — | — | — | — | — | |
| SINK + TAIL + TOP-KModel Backbone=QWEN3-1.7B, Load Budget=5%2026.04 | 85.3 | — | — | — | — | — | — | — | — | — | |
| SINK + TAIL + TOP-KModel Backbone=QWEN3-1.7B, Load Budget=3%2026.04 | 85 | — | — | — | — | — | — | — | — | — | |
| SINK + TAIL + TOP-K + ϕModel Backbone=QWEN3-1.7B, Load Budget=5%2026.04 | 84.8 | — | — | — | — | — | — | — | — | — | |
| SINK + TAIL + TOP-K + ϕModel Backbone=QWEN3-1.7B, Load Budget=4%2026.04 | 84.7 | — | — | — | — | — | — | — | — | — | |
| SINK + TAIL + TOP-KModel Backbone=QWEN3-1.7B, Load Budget=2%2026.04 | 84.5 | — | — | — | — | — | — | — | — | — | |
| SINK + TAIL + TOP-K + ϕModel Backbone=QWEN3-1.7B, Load Budget=3%2026.04 | 84.4 | — | — | — | — | — | — | — | — | — | |
| SINK + TAIL + TOP-K + ϕModel Backbone=QWEN3-1.7B, Load Budget=2%2026.04 | 84.1 | — | — | — | — | — | — | — | — | — | |
| MAGICPIGModel Backbone=QWEN3-1.7B, Load Budget=3%2026.04 | 81.7 | — | — | — | — | — | — | — | — | — | |
| SnapKVContext Length=8K2026.03 | 81.03 | 100 | 98.2 | 96.8 | 92 | 75.95 | 71.9 | 80.52 | 63.93 | 50 | |
| PyramidKVContext Length=8K2026.03 | 80.99 | 100 | 98 | 96.8 | 93 | 74.6 | 72.05 | 84.16 | 60.28 | 50 | |
| MAGICPIGModel Backbone=QWEN3-1.7B, Load Budget=4%2026.04 | 80.8 | — | — | — | — | — | — | — | — | — | |
| MAGICPIGModel Backbone=QWEN3-1.7B, Load Budget=5%2026.04 | 79.7 | — | — | — | — | — | — | — | — | — | |
| Full (exact)Model Backbone=LLAMA-3.2-1B-INSTRUCT, Load Budget=100% (Full)2026.04 | 74.3 | — | — | — | — | — | — | — | — | — | |
| MAGICPIGModel Backbone=QWEN3-1.7B, Load Budget=2%2026.04 | 74.3 | — | — | — | — | — | — | — | — | — | |
| SINK + TAIL + TOP-KModel Backbone=LLAMA-3.2-1B-INSTRUCT, Load Budget=5%2026.04 | 73.6 | — | — | — | — | — | — | — | — | — | |
| SINK + TAIL + TOP-K + ϕModel Backbone=LLAMA-3.2-1B-INSTRUCT, Load Budget=5%2026.04 | 73.5 | — | — | — | — | — | — | — | — | — | |
| SINK + TAIL + TOP-K + ϕModel Backbone=LLAMA-3.2-1B-INSTRUCT, Load Budget=4%2026.04 | 73.4 | — | — | — | — | — | — | — | — | — | |
| SINK + TAIL + TOP-KModel Backbone=LLAMA-3.2-1B-INSTRUCT, Load Budget=4%2026.04 | 73.2 | — | — | — | — | — | — | — | — | — | |
| SINK + TAIL + TOP-K + ϕModel Backbone=LLAMA-3.2-1B-INSTRUCT, Load Budget=3%2026.04 | 73 | — | — | — | — | — | — | — | — | — | |
| SINK + TAIL + TOP-KModel Backbone=LLAMA-3.2-1B-INSTRUCT, Load Budget=3%2026.04 | 72.7 | — | — | — | — | — | — | — | — | — | |
| SINK + TAIL + TOP-K + ϕModel Backbone=LLAMA-3.2-1B-INSTRUCT, Load Budget=2%2026.04 | 72.5 | — | — | — | — | — | — | — | — | — | |
| SINK + TAIL + TOP-KModel Backbone=LLAMA-3.2-1B-INSTRUCT, Load Budget=2%2026.04 | 71.4 | — | — | — | — | — | — | — | — | — | |
| SINK + TAIL + TOP-K + ϕModel Backbone=LLAMA-3.2-1B-INSTRUCT, Load Budget=1%2026.04 | 71.4 | — | — | — | — | — | — | — | — | — | |
| SINK + TAIL + TOP-KModel Backbone=LLAMA-3.2-1B-INSTRUCT, Load Budget=1%2026.04 | 69.1 | — | — | — | — | — | — | — | — | — | |
| MAGICPIGModel Backbone=LLAMA-3.2-1B-INSTRUCT, Load Budget=3%2026.04 | 66.8 | — | — | — | — | — | — | — | — | — | |
| MAGICPIGModel Backbone=LLAMA-3.2-1B-INSTRUCT, Load Budget=4%2026.04 | 64.2 | — | — | — | — | — | — | — | — | — | |
| MAGICPIGModel Backbone=LLAMA-3.2-1B-INSTRUCT, Load Budget=5%2026.04 | 62.5 | — | — | — | — | — | — | — | — | — | |
| QUESTModel Backbone=QWEN3-1.7B, Load Budget=5%2026.04 | 60.4 | — | — | — | — | — | — | — | — | — | |
| QUESTModel Backbone=QWEN3-1.7B, Load Budget=4%2026.04 | 57.4 | — | — | — | — | — | — | — | — | — | |
| MAGICPIGModel Backbone=LLAMA-3.2-1B-INSTRUCT, Load Budget=2%2026.04 | 55.8 | — | — | — | — | — | — | — | — | — | |
| QUESTModel Backbone=QWEN3-1.7B, Load Budget=3%2026.04 | 51.3 | — | — | — | — | — | — | — | — | — | |
| QUESTModel Backbone=LLAMA-3.2-1B-INSTRUCT, Load Budget=5%2026.04 | 51.2 | — | — | — | — | — | — | — | — | — | |
| MAGICPIGModel Backbone=LLAMA-3.2-1B-INSTRUCT, Load Budget=1%2026.04 | 49 | — | — | — | — | — | — | — | — | — | |
| SNAPKVModel Backbone=QWEN3-1.7B, Load Budget=5%2026.04 | 48.8 | — | — | — | — | — | — | — | — | — | |
| QUESTModel Backbone=LLAMA-3.2-1B-INSTRUCT, Load Budget=4%2026.04 | 48.7 | — | — | — | — | — | — | — | — | — | |
| SNAPKVModel Backbone=QWEN3-1.7B, Load Budget=4%2026.04 | 46.5 | — | — | — | — | — | — | — | — | — | |
| QUESTModel Backbone=LLAMA-3.2-1B-INSTRUCT, Load Budget=3%2026.04 | 44.4 | — | — | — | — | — | — | — | — | — | |
| PYRAMIDKVModel Backbone=QWEN3-1.7B, Load Budget=5%2026.04 | 42.9 | — | — | — | — | — | — | — | — | — | |
| SNAPKVModel Backbone=LLAMA-3.2-1B-INSTRUCT, Load Budget=5%2026.04 | 42.7 | — | — | — | — | — | — | — | — | — | |
| QUESTModel Backbone=QWEN3-1.7B, Load Budget=2%2026.04 | 42.3 | — | — | — | — | — | — | — | — | — | |
| SNAPKVModel Backbone=QWEN3-1.7B, Load Budget=3%2026.04 | 41.1 | — | — | — | — | — | — | — | — | — | |
| SNAPKVModel Backbone=LLAMA-3.2-1B-INSTRUCT, Load Budget=4%2026.04 | 40.3 | — | — | — | — | — | — | — | — | — | |
| PYRAMIDKVModel Backbone=QWEN3-1.7B, Load Budget=4%2026.04 | 39.2 | — | — | — | — | — | — | — | — | — | |
| QUESTModel Backbone=LLAMA-3.2-1B-INSTRUCT, Load Budget=2%2026.04 | 38 | — | — | — | — | — | — | — | — | — | |
| SNAPKVModel Backbone=LLAMA-3.2-1B-INSTRUCT, Load Budget=3%2026.04 | 35.8 | — | — | — | — | — | — | — | — | — | |
| PYRAMIDKVModel Backbone=LLAMA-3.2-1B-INSTRUCT, Load Budget=5%2026.04 | 34.1 | — | — | — | — | — | — | — | — | — | |
| PYRAMIDKVModel Backbone=QWEN3-1.7B, Load Budget=3%2026.04 | 33 | — | — | — | — | — | — | — | — | — | |
| SNAPKVModel Backbone=QWEN3-1.7B, Load Budget=2%2026.04 | 32.2 | — | — | — | — | — | — | — | — | — | |
| PYRAMIDKVModel Backbone=LLAMA-3.2-1B-INSTRUCT, Load Budget=4%2026.04 | 31.1 | — | — | — | — | — | — | — | — | — | |
| SNAPKVModel Backbone=LLAMA-3.2-1B-INSTRUCT, Load Budget=2%2026.04 | 29.3 | — | — | — | — | — | — | — | — | — | |
| QUESTModel Backbone=LLAMA-3.2-1B-INSTRUCT, Load Budget=1%2026.04 | 25.4 | — | — | — | — | — | — | — | — | — | |
| PYRAMIDKVModel Backbone=LLAMA-3.2-1B-INSTRUCT, Load Budget=3%2026.04 | 23.9 | — | — | — | — | — | — | — | — | — | |
| PYRAMIDKVModel Backbone=QWEN3-1.7B, Load Budget=2%2026.04 | 20.2 | — | — | — | — | — | — | — | — | — | |
| SNAPKVModel Backbone=LLAMA-3.2-1B-INSTRUCT, Load Budget=1%2026.04 | 16.6 | — | — | — | — | — | — | — | — | — | |
| PYRAMIDKVModel Backbone=LLAMA-3.2-1B-INSTRUCT, Load Budget=2%2026.04 | 13.2 | — | — | — | — | — | — | — | — | — | |
| H2OContext Length=8K2026.03 | 10.12 | 1.8 | 0.2 | 0.4 | 3 | 0.05 | 0.3 | 15.76 | 36.8 | 32.8 | |
| STREAMINGLLMModel Backbone=QWEN3-1.7B, Load Budget=5%2026.04 | 8.8 | — | — | — | — | — | — | — | — | — | |
| STREAMINGLLMModel Backbone=LLAMA-3.2-1B-INSTRUCT, Load Budget=5%2026.04 | 7.5 | — | — | — | — | — | — | — | — | — | |
| STREAMINGLLMModel Backbone=QWEN3-1.7B, Load Budget=4%2026.04 | 6.8 | — | — | — | — | — | — | — | — | — | |
| STREAMINGLLMModel Backbone=LLAMA-3.2-1B-INSTRUCT, Load Budget=4%2026.04 | 6.2 | — | — | — | — | — | — | — | — | — | |
| PYRAMIDKVModel Backbone=LLAMA-3.2-1B-INSTRUCT, Load Budget=1%2026.04 | 5 | — | — | — | — | — | — | — | — | — | |
| STREAMINGLLMModel Backbone=LLAMA-3.2-1B-INSTRUCT, Load Budget=3%2026.04 | 5 | — | — | — | — | — | — | — | — | — | |
| STREAMINGLLMModel Backbone=LLAMA-3.2-1B-INSTRUCT, Load Budget=2%2026.04 | 4.6 | — | — | — | — | — | — | — | — | — | |
| STREAMINGLLMModel Backbone=QWEN3-1.7B, Load Budget=3%2026.04 | 4.6 | — | — | — | — | — | — | — | — | — | |
| STREAMINGLLMModel Backbone=LLAMA-3.2-1B-INSTRUCT, Load Budget=1%2026.04 | 4.3 | — | — | — | — | — | — | — | — | — | |
| STREAMINGLLMModel Backbone=QWEN3-1.7B, Load Budget=2%2026.04 | 3.7 | — | — | — | — | — | — | — | — | — |