Long-context evaluation on RULER 32K context length (test)
100Niah1 Scorefp16 Baseline
Evaluation results
| Method | Links | ||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| fp16 BaselineAvg. bit=162024.01 | 100 | 99.8 | 98.6 | 94 | 68.2 | 11 | 55.95 | 64.5 | 37.88 | 9.64 | 30.4 | 31.6 | 31.6 | 56.4 | |
| QuestBackbone Model=Llama-2-7B-32K-Instruct, Context Length=32K, Token Budget=1024 (3.13%)2025.06 | 100 | 95.83 | 52.08 | 87.5 | 54.17 | — | 52.34 | 56.51 | 26.87 | — | 36.36 | 23.96 | 34.38 | 56.37 | |
| HATABackbone Model=Llama-2-7B-32K-Instruct, Context Length=32K, Token Budget=1024 (3.13%)2025.06 | 100 | 98.96 | 83.33 | 93.75 | 78.12 | — | 65.62 | 54.17 | 20 | — | 43.4 | 28.12 | 37.5 | 63.91 | |
| KVQuant-3bit-1%Avg. bit=3.332024.01 | 99.8 | 98.8 | 95.2 | 92.8 | 61.6 | 6.4 | 47.5 | 54.45 | 41.04 | 8.52 | 29.33 | 31 | 31 | 53.65 | |
| MagicPIGBackbone Model=Llama-2-7B-32K-Instruct, Context Length=32K, Token Budget=2-3%2025.06 | 97.92 | 93.75 | 54.17 | 83.33 | 71.88 | — | 59.38 | 43.49 | 16.04 | — | 51.39 | 30.21 | 35.42 | 57.91 | |
| KVQuant-2bit-1%Avg. bit=2.332024.01 | 95.4 | 86.8 | 49.8 | 73.6 | 23.4 | 0 | 16.65 | 22.95 | 22.52 | 5.14 | 24 | 26.4 | 28.4 | 36.54 | |
| DenseBackbone Model=Llama-2-7B-32K-Instruct, Context Length=32K, Token Budget=Full Attention2025.06 | 93.75 | 100 | 91.67 | 93.75 | 81.25 | — | 66.67 | 52.08 | 21.04 | — | 48.61 | 30.21 | 36.46 | 65.04 | |
| KIVI-2-gs32-r128Avg. bit=3.05, group size=32, residual=128-element fp162024.01 | 76 | 85.6 | 59.6 | 72.6 | 11.4 | 0 | 34.7 | 46.45 | 39.6 | 8.26 | 30.53 | 24.8 | 27.6 | 39.78 | |
| LokiBackbone Model=Llama-2-7B-32K-Instruct, Context Length=32K, Token Budget=1024 (3.13%)2025.06 | 25 | 2.08 | 0 | 0 | 0 | — | 0 | 0 | 1.04 | — | 19.44 | 14.58 | 16.67 | 7.16 | |
| SnapKVBackbone Model=Llama-2-7B-32K-Instruct, Context Length=32K, Token Budget=1024 (3.13%)2025.06 | 25 | 1.04 | 0 | 0 | 0 | — | 0.26 | 0 | 3.96 | — | 20.49 | 22.92 | 26.04 | 9.25 | |
| StreamingLLMBackbone Model=Llama-2-7B-32K-Instruct, Context Length=32K, Token Budget=1024 (3.13%)2025.06 | 1.04 | 5.21 | 1.04 | 6.25 | 1.04 | — | 6.25 | 4.17 | 0.21 | — | 54.86 | 22.92 | 27.08 | 11.82 | |
| H2OBackbone Model=Llama-2-7B-32K-Instruct, Context Length=32K, Token Budget=1024 (3.13%)2025.06 | 0 | 0 | 0 | 3.12 | 0 | — | 0.78 | 0 | 0 | — | 20.49 | 15.62 | 16.67 | 5.15 |