Long-context language modeling on LongBench (test)
50.87Qasper ScoreFull-context
Evaluation results
| Method | Links | |||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Full-contextPrefill=100%, KV=100%2026.04 | 50.87 | 30.21 | 31.89 | 79.82 | 92.74 | 53.15 | 69.87 | 75.43 | — | 58.43 | — | — | 38.65 | 38.76 | 17.56 | 100 | 55.82 | 34.21 | 67.54 | 53.92 | — | |
| SnapKVPrefill=100%, KV=20%2026.04 | 50.23 | 29.34 | 30.56 | 78.45 | 92.12 | 52.45 | 68.98 | 66.54 | — | 56.78 | — | — | 36.23 | 37.98 | 16.12 | 99.5 | 54.05 | 33.12 | 63.56 | 52.89 | — | |
| StructKVPrefill=60%, KV=20%2026.04 | 50.15 | 30.12 | 31.34 | 79.43 | 92.66 | 53.01 | 68.72 | 74.56 | — | 57.67 | — | — | 37.42 | 38.56 | 16.67 | 100 | 55.19 | 34.12 | 64.89 | 53.67 | — | |
| FastKVPrefill=60%, KV=20%2026.04 | 50.12 | 29.67 | 30.78 | 78.89 | 92.12 | 52.34 | 68.45 | 73.12 | — | 57.23 | — | — | 36.45 | 38.34 | 15.98 | 99.5 | 54.49 | 33.56 | 63.23 | 52.12 | — | |
| StreamingLLMPrefill=100%, KV=20%2026.04 | 49.87 | 28.45 | 30.23 | 78.12 | 92.15 | 51.67 | 67.98 | 66.87 | — | 55.43 | — | — | 33.56 | 37.98 | 14.32 | 90.5 | 52.57 | 32.56 | 59.12 | 52.34 | — | |
| FastKVPrefill=60%, KV=10%2026.04 | 49.85 | 28.92 | 30.15 | 77.45 | 91.56 | 51.89 | 67.89 | 72.12 | — | 56.78 | — | — | 35.67 | 37.89 | 15.43 | 99.5 | 53.88 | 33.12 | 62.45 | 51.34 | — | |
| SnapKVPrefill=100%, KV=10%2026.04 | 49.76 | 28.65 | 30.23 | 77.23 | 91.56 | 52.12 | 68.12 | 65.67 | — | 55.98 | — | — | 35.12 | 37.56 | 15.34 | 99.5 | 53.46 | 32.87 | 63.12 | 52.45 | — | |
| StructKVPrefill=60%, KV=10%2026.04 | 49.65 | 29.87 | 31.23 | 79.12 | 92.45 | 52.78 | 68.56 | 73.34 | — | 56.34 | — | — | 36.78 | 38.34 | 15.83 | 100 | 54.62 | 33.15 | 64.21 | 52.24 | — | |
| StreamingLLMPrefill=100%, KV=10%2026.04 | 49.12 | 27.89 | 29.56 | 77.34 | 91.56 | 50.89 | 66.78 | 65.43 | — | 54.37 | — | — | 32.18 | 37.15 | 13.45 | 90.5 | 51.62 | 30.84 | 57.65 | 51.23 | — | |
| GemFilterPrefill=70%, KV=20%2026.04 | 49.12 | 27.87 | 29.34 | 77.45 | 91.89 | 48.78 | 66.12 | 55.67 | — | 51.23 | — | — | 34.87 | 36.98 | 12.12 | 96.5 | 51.01 | 29.89 | 57.12 | 51.23 | — | |
| GemFilterPrefill=60%, KV=10%2026.04 | 48.34 | 27.12 | 28.78 | 76.23 | 91.45 | 48.12 | 65.34 | 53.23 | — | 49.87 | — | — | 33.45 | 36.23 | 10.87 | 94.5 | 49.94 | 29.12 | 56.23 | 50.12 | — | |
| Full CacheCache Budget (B)=Full, Model=Llama-3.1-8B-Instruct2026.06 | 45.37 | 25.38 | 27.2 | 72.5 | 91.64 | 43.57 | 62.88 | 56.43 | — | 55.8 | — | — | 31.26 | — | 9.41 | 99.5 | 49.2 | 30.22 | 55.97 | 45 | 35.12 | |
| K-VECCache Budget (B)=1024, Model=Llama-3.1-8B-Instruct2026.06 | 45.03 | 24.99 | 26.04 | 71.49 | 91.72 | 42.69 | 62.99 | 56.55 | — | 56.64 | — | — | 31.39 | — | 8.45 | 99.5 | 48.92 | 30.04 | 58.2 | 48.09 | 28.98 | |
| Ada-PyramidCache Budget (B)=1024, Model=Llama-3.1-8B-Instruct2026.06 | 44.57 | 25.26 | 25.84 | 70.5 | 91.62 | 42.37 | 62.96 | 56.52 | — | 53.73 | — | — | 31.97 | — | 7.67 | 99.5 | 48 | 28.76 | 54.89 | 44.15 | 27.75 | |
| Ada-SnapKVCache Budget (B)=1024, Model=Llama-3.1-8B-Instruct2026.06 | 44.09 | 24.73 | 26.04 | 72.5 | 91.72 | 42.56 | 63.22 | 56.33 | — | 53.82 | — | — | 31.4 | — | 7.82 | 99.5 | 48.16 | 29.23 | 54.8 | 44.01 | 28.86 | |
| PyramidCache Budget (B)=1024, Model=Llama-3.1-8B-Instruct2026.06 | 43.66 | 24.3 | 25.68 | 68.5 | 91.27 | 41.96 | 63.13 | 55.85 | — | 54.1 | — | — | 31.3 | — | 7.73 | 99.5 | 47.7 | 29.62 | 55.06 | 44.22 | 27.27 | |
| SnapKVCache Budget (B)=1024, Model=Llama-3.1-8B-Instruct2026.06 | 43.64 | 24.39 | 25.95 | 69 | 91.72 | 42.5 | 62.99 | 56.45 | — | 54.34 | — | — | 31.52 | — | 7.8 | 99.5 | 47.84 | 29.28 | 54.24 | 44.34 | 27.8 | |
| K-VECCache Budget (B)=512, Model=Llama-3.1-8B-Instruct2026.06 | 42.81 | 24.86 | 24.92 | 67.47 | 91.62 | 42.67 | 62.9 | 55.68 | — | 56.07 | — | — | 31.39 | — | 8.45 | 99.5 | 48.12 | 30.04 | 57.9 | 47.39 | 26.24 | |
| PyramidInferPrefill=60%, KV=60%2026.04 | 42.76 | 25.98 | 23.89 | 73.23 | 90.56 | 49.87 | 66.23 | 65.78 | — | 46.54 | — | — | 28.23 | 31.87 | 9.12 | 88.5 | 48.06 | 29.87 | 49.12 | 47.34 | — | |
| FullBackbone=Mistral-7B-Instruct, KV size=2562026.03 | 41.54 | 25.71 | 27.73 | 76 | 88.59 | 47.51 | 61.48 | 62.68 | 29.07 | 52.88 | 49.37 | 39.01 | 28.58 | 35.07 | 6 | 98.5 | 48.11 | — | — | — | — | |
| MixedDimKVBackbone=Mistral-7B-Instruct, KV size=2562026.03 | 41.5 | 25.33 | 27.59 | 75 | 88.75 | 47.81 | 61.44 | 62.3 | 29.62 | 53.05 | 48.49 | 38.7 | 28.16 | 30.67 | 6 | 98 | 47.65 | — | — | — | — | |
| Ada-PyramidCache Budget (B)=512, Model=Llama-3.1-8B-Instruct2026.06 | 40.63 | 24.12 | 24.61 | 69 | 91.79 | 42.55 | 62.28 | 54.49 | — | 53.03 | — | — | 30.26 | — | 7.95 | 99.5 | 46.98 | 28.04 | 54.71 | 43.39 | 25.35 | |
| K-VECCache Budget (B)=256, Model=Llama-3.1-8B-Instruct2026.06 | 40.38 | 23.94 | 22.95 | 62.74 | 91.89 | 41.95 | 60.87 | 54.9 | — | 53.53 | — | — | 31.24 | — | 8.05 | 99.5 | 46.7 | 26.33 | 57.39 | 47.14 | 24.35 | |
| Ada-SnapKVCache Budget (B)=512, Model=Llama-3.1-8B-Instruct2026.06 | 40.16 | 24.39 | 24.85 | 69 | 92.34 | 42.05 | 63.43 | 55.32 | — | 52.44 | — | — | 31.1 | — | 7.98 | 99.5 | 47.15 | 29.07 | 53.9 | 43.05 | 25.75 | |
| SnapKVCache Budget (B)=512, Model=Llama-3.1-8B-Instruct2026.06 | 40.01 | 23.77 | 24.19 | 64 | 92.05 | 41.57 | 63.21 | 55.05 | — | 53.15 | — | — | 31.32 | — | 8.01 | 99.5 | 46.76 | 29.22 | 54.47 | 43.63 | 25.04 | |
| PyramidCache Budget (B)=512, Model=Llama-3.1-8B-Instruct2026.06 | 39.89 | 23.73 | 24.28 | 66 | 91.07 | 41.42 | 61.99 | 53.44 | — | 52.86 | — | — | 31.22 | — | 8.39 | 99.5 | 46.59 | 28.71 | 54 | 44.2 | 24.74 | |
| K-VECCache Budget (B)=128, Model=Llama-3.1-8B-Instruct2026.06 | 35.96 | 23.47 | 21.88 | 52.89 | 90.35 | 41.25 | 59.35 | 51.25 | — | 50.4 | — | — | 28.75 | — | 7.85 | 99.5 | 44.57 | 25.96 | 54.79 | 47.14 | 22.35 | |
| Ada-PyramidCache Budget (B)=256, Model=Llama-3.1-8B-Instruct2026.06 | 35.06 | 23.36 | 22.67 | 63 | 90.72 | 41.21 | 61.47 | 53.09 | — | 52.28 | — | — | 28.76 | — | 7.75 | 99.5 | 45.23 | 25.12 | 54.66 | 41.89 | 23.14 | |
| H2OBackbone=Mistral-7B-Instruct, KV size=2562026.03 | 34.24 | 21.61 | 25.89 | 72.5 | 87.69 | 40.96 | 53.94 | 47.76 | 25.93 | 42.34 | 46.22 | 33.65 | 22.8 | 26.9 | 6.5 | 88 | 42.31 | — | — | — | — | |
| Ada-SnapKVCache Budget (B)=256, Model=Llama-3.1-8B-Instruct2026.06 | 33.39 | 23.85 | 22.88 | 63.5 | 91.57 | 40.94 | 61.95 | 54.33 | — | 51.44 | — | — | 29.54 | — | 8 | 99.5 | 45.44 | 26.11 | 54.94 | 42.15 | 23.01 | |
| PyramidBackbone=Mistral-7B-Instruct, KV size=2562026.03 | 33.36 | 23.2 | 23.25 | 72.5 | 89.56 | 45.14 | 56.9 | 55.62 | 27.6 | 52.23 | 48.48 | 38.11 | 27.1 | 23.64 | 5.5 | 96 | 44.89 | — | — | — | — | |
| SnapKVBackbone=Mistral-7B-Instruct, KV size=2562026.03 | 33.18 | 23.12 | 23.21 | 73 | 88.79 | 44.88 | 58.05 | 58.12 | 27.72 | 51.59 | 47.58 | 37.6 | 27.73 | 23.98 | 5 | 96 | 44.97 | — | — | — | — | |
| PyramidCache Budget (B)=256, Model=Llama-3.1-8B-Instruct2026.06 | 33.15 | 23.37 | 22.33 | 56.5 | 91.19 | 41.28 | 60.36 | 51.18 | — | 51.44 | — | — | 28.62 | — | 6.97 | 99.5 | 44.47 | 25.53 | 55.03 | 42.42 | 22.57 | |
| SnapKVCache Budget (B)=256, Model=Llama-3.1-8B-Instruct2026.06 | 32.49 | 23.54 | 22.55 | 53.5 | 91.1 | 40.95 | 60.67 | 53.39 | — | 51.62 | — | — | 28.94 | — | 7.48 | 99.5 | 44.48 | 26.02 | 54.4 | 42.77 | 22.83 | |
| FullBackbone=Llama-3-8B-Instruct, KV size=2562026.03 | 32.3 | 23.22 | 26.67 | 74 | 90.48 | 42.29 | 59.13 | 54.02 | 25.56 | 39.71 | 43.56 | 35.49 | 21.14 | 28.58 | 4.8 | 69.75 | 41.92 | — | — | — | — | |
| MixedDimKVBackbone=Llama-3-8B-Instruct, KV size=2562026.03 | 31.39 | 22.95 | 26.54 | 73 | 90.61 | 42.4 | 59.11 | 54.84 | 25.26 | 39.6 | 43.82 | 35.46 | 21.59 | 27.07 | 5.02 | 69.5 | 41.76 | — | — | — | — | |
| SLMCache Budget (B)=1024, Model=Llama-3.1-8B-Instruct2026.06 | 30.22 | 21.08 | 25.72 | 63.5 | 88.87 | 42.28 | 61.3 | 53.4 | — | 37.06 | — | — | 25.24 | — | 6.98 | 89 | 42.58 | 24.97 | 46.57 | 39.14 | 26.01 | |
| SLMCache Budget (B)=512, Model=Llama-3.1-8B-Instruct2026.06 | 25.78 | 21.26 | 23.57 | 57.5 | 87.86 | 41.44 | 60.85 | 51.02 | — | 34.19 | — | — | 24.93 | — | 6.98 | 96.5 | 41.37 | 25.51 | 45.01 | 35.91 | 23.61 | |
| Ada-PyramidCache Budget (B)=128, Model=Llama-3.1-8B-Instruct2026.06 | 25.61 | 23.18 | 20.38 | 51.5 | 90.76 | 40.62 | 59.3 | 50.88 | — | 49.3 | — | — | 27.83 | — | 6.92 | 99 | 42.96 | 27.07 | 53.02 | 41.29 | 20.7 | |
| PyramidCache Budget (B)=128, Model=Llama-3.1-8B-Instruct2026.06 | 24.69 | 22.38 | 19.74 | 44.5 | 88.81 | 40.3 | 57.25 | 49.9 | — | 47.74 | — | — | 27.23 | — | 7.22 | 99.5 | 41.78 | 25.7 | 52.87 | 40.57 | 20.02 | |
| SnapKVCache Budget (B)=128, Model=Llama-3.1-8B-Instruct2026.06 | 24.45 | 22.36 | 19.55 | 45.5 | 89.2 | 40.62 | 58.45 | 49.9 | — | 48.03 | — | — | 28.19 | — | 6.97 | 99.5 | 42.03 | 25.54 | 53.31 | 40.75 | 20.13 | |
| Ada-SnapKVCache Budget (B)=128, Model=Llama-3.1-8B-Instruct2026.06 | 24.41 | 23.21 | 20.28 | 50.5 | 89.49 | 40.71 | 58.74 | 52.4 | — | 49.95 | — | — | 28.55 | — | 7.45 | 99 | 42.81 | 24.9 | 53.15 | 41.73 | 20.54 | |
| SLMCache Budget (B)=256, Model=Llama-3.1-8B-Instruct2026.06 | 23.79 | 20.68 | 20.67 | 46 | 87.11 | 40.82 | 59.89 | 49.19 | — | 31.8 | — | — | 25.55 | — | 7.2 | 99.5 | 39.76 | 22.71 | 43.43 | 36.55 | 21.29 | |
| H2OPrefill=100%, KV=20%2026.04 | 22.12 | 26.89 | 29.87 | 74.87 | 93.12 | 49.65 | 66.23 | 62.45 | — | 35.23 | — | — | 20.87 | 36.56 | 11.89 | 85.5 | 43.56 | 28.98 | 26.54 | 26.12 | — | |
| SLMCache Budget (B)=128, Model=Llama-3.1-8B-Instruct2026.06 | 20.87 | 21.09 | 18.48 | 40.5 | 84.41 | 38.82 | 57.02 | 47.29 | — | 31.72 | — | — | 24.54 | — | 8 | 99.5 | 38.43 | 22.24 | 44.02 | 37.55 | 18.76 | |
| PyramidBackbone=Llama-3-8B-Instruct, KV size=2562026.03 | 20.51 | 22.68 | 22.83 | 71 | 90.48 | 39.86 | 58.64 | 54.06 | 23.99 | 36.06 | 42.47 | 31.34 | 20.28 | 21.36 | 5.83 | 69.25 | 39.42 | — | — | — | — | |
| SnapKVBackbone=Llama-3-8B-Instruct, KV size=2562026.03 | 20.2 | 22 | 22.83 | 71 | 90.86 | 40.03 | 60.16 | 55.91 | 23.36 | 37.36 | 42.37 | 33.18 | 19.96 | 21.76 | 5.83 | 69.5 | 39.77 | — | — | — | — | |
| H2OPrefill=100%, KV=10%2026.04 | 19.34 | 27.45 | 30.65 | 74.23 | 92.65 | 48.98 | 64.12 | 61.23 | — | 32.67 | — | — | 16.43 | 36.87 | 11.23 | 82.5 | 42 | 27.56 | 23.45 | 22.56 | — | |
| H2OBackbone=Llama-3-8B-Instruct, KV size=2562026.03 | 18.56 | 21.48 | 24.57 | 60.5 | 88.7 | 38.55 | 58.52 | 52.73 | 23.09 | 29.02 | 36.01 | 28.07 | 15.7 | 23.31 | 5.37 | 68.04 | 37.01 | — | — | — | — | |
| MoQAE2025.06 | 9.79 | 21.23 | 3.47 | 66 | 87.89 | 41.37 | 66.53 | 59.94 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| CQ-4c8bchannels per group=4, quantization bits=82025.06 | 9.58 | 20.87 | 1.93 | 66 | 87.72 | 41.13 | 66.57 | 59.75 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| FP16precision=full precision (FP16)2025.06 | 9.52 | 21.28 | 3.51 | 66 | 87.72 | 41.69 | 66.66 | 59.82 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| KIVI-2bquantization bits=2-bit2025.06 | 9.26 | 20.53 | 0.97 | 66 | 87.42 | 42.61 | 66.22 | 59.67 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| MiKVquantization_type=mixed-precision2025.06 | 9.14 | 20.63 | 0.85 | 65.88 | 87.21 | 41.44 | 66.18 | 59.55 | — | — | — | — | — | — | — | — | — | — | — | — | — |