Long-context understanding on RULER (14-Metric Evaluation)
100VT ScoreFullKV
Evaluation results
| Method | Links | ||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| FullKVBackbone=Meta-Llama-3.1-8B-Instruct, Cache Budget=20% Cache2026.05 | 100 | — | — | — | — | — | — | — | — | 51 | 94.67 | 84 | 54 | 90.82 | 100 | 100 | 100 | 97.5 | 99.5 | 100 | 100 | 100 | |
| Full KVModel=Llama3.1-8B-Instruct, Compression Ratio=1.02026.03 | 99.2 | 100 | 100 | 100 | 99 | 99 | 99 | 98.75 | 98.5 | 17.8 | 87 | 87 | 54 | 87.63 | — | — | — | — | — | — | — | — | |
| LaProxBackbone=Meta-Llama-3.1-8B-Instruct, Cache Budget=20% Cache2026.05 | 99.2 | — | — | — | — | — | — | — | — | 12 | 90.33 | 65 | 54 | 85.21 | 100 | 99 | 97 | 97.5 | 96.75 | 100 | 100 | 97 | |
| EchoKV-HybridModel=Llama3.1-8B-Instruct, Compression Ratio=0.52026.03 | 97.4 | 100 | 100 | 100 | 99 | 99 | 99 | 97 | 100 | 13.8 | 79.67 | 85 | 54 | 86.45 | — | — | — | — | — | — | — | — | |
| Full KVModel=Mistral-7B-Instruct-v0.3, Compression Ratio=1.02026.03 | 95.8 | 100 | 92 | 100 | 84 | 80 | 57 | 92.25 | 92.25 | 80.7 | 89.67 | 65 | 47 | 82.74 | — | — | — | — | — | — | — | — | |
| AdaKVBackbone=Meta-Llama-3.1-8B-Instruct, Cache Budget=20% Cache2026.05 | 92.2 | — | — | — | — | — | — | — | — | 10 | 85.33 | 34 | 47 | 57.38 | 82 | 21 | 25 | 74 | 62.5 | 98 | 94 | 21 | |
| EchoKV-HybridModel=Mistral-7B-Instruct-v0.3, Compression Ratio=0.52026.03 | 90.8 | 100 | 89 | 100 | 83 | 77 | 42 | 92.5 | 90.25 | 66.3 | 93.67 | 67 | 47 | 79.89 | — | — | — | — | — | — | — | — | |
| CriticalKVBackbone=Meta-Llama-3.1-8B-Instruct, Cache Budget=20% Cache2026.05 | 90 | — | — | — | — | — | — | — | — | 12.6 | 75.33 | 44 | 48 | 58.26 | 88 | 14 | 5 | 86.25 | 77.25 | 99 | 100 | 18 | |
| EchoKVModel=Llama3.1-8B-Instruct, Compression Ratio=0.52026.03 | 87.2 | 100 | 100 | 100 | 99 | 99 | 94 | 89.5 | 99.5 | 0.6 | 80 | 85 | 52 | 83.52 | — | — | — | — | — | — | — | — | |
| PaluModel=Llama3.1-8B-Instruct, Compression Ratio=0.52026.03 | 86 | 100 | 98 | 75 | 97 | 78 | 30 | 73 | 64.25 | 15.3 | 78.33 | 53 | 38 | 68.14 | — | — | — | — | — | — | — | — | |
| EchoKVModel=Mistral-7B-Instruct-v0.3, Compression Ratio=0.52026.03 | 85.2 | 100 | 88 | 99 | 82 | 59 | 19 | 62.75 | 67.5 | 46.6 | 81.67 | 65 | 43 | 69.13 | — | — | — | — | — | — | — | — | |
| PaluModel=Mistral-7B-Instruct-v0.3, Compression Ratio=0.52026.03 | 83 | 100 | 97 | 98 | 76 | 51 | 26 | 88.5 | 80.75 | 32.1 | 85 | 41 | 33 | 68.56 | — | — | — | — | — | — | — | — | |
| EchoKV-HybridModel=Mistral-7B-Instruct-v0.3, Compression Ratio=0.32026.03 | 80 | 99 | 80 | 88 | 76 | 53 | 10 | 48 | 50.75 | 27.1 | 82.33 | 61 | 45 | 61.55 | — | — | — | — | — | — | — | — | |
| CommonKVModel=Llama3.1-8B-Instruct, Compression Ratio=0.52026.03 | 79.8 | 92 | 94 | 83 | 97 | 67 | 1 | 87 | 94.25 | 0.1 | 85.67 | 75 | 50 | 69.68 | — | — | — | — | — | — | — | — | |
| CommonKVModel=Mistral-7B-Instruct-v0.3, Compression Ratio=0.52026.03 | 66.6 | 86 | 46 | 36 | 27 | 18 | 0 | 7 | 12.75 | 15.1 | 75 | 46 | 44 | 36.88 | — | — | — | — | — | — | — | — | |
| ThinK*Model=Mistral-7B-Instruct-v0.3, Compression Ratio=0.62026.03 | 44.2 | 21 | 58 | 2 | 32 | 17 | 5 | 20.25 | 14 | 17.1 | 89.67 | 60 | 42 | 32.47 | — | — | — | — | — | — | — | — | |
| EchoKVModel=Mistral-7B-Instruct-v0.3, Compression Ratio=0.32026.03 | 39.8 | 77 | 64 | 28 | 54 | 14 | 1 | 6 | 9 | 15.3 | 51 | 56 | 41 | 35.08 | — | — | — | — | — | — | — | — | |
| CommonKVModel=Llama3.1-8B-Instruct, Compression Ratio=0.32026.03 | 38.4 | 45 | 28 | 0 | 13 | 0 | 0 | 7.5 | 8.5 | 9.9 | 39.33 | 21 | 30 | 18.51 | — | — | — | — | — | — | — | — | |
| EchoKVModel=Llama3.1-8B-Instruct, Compression Ratio=0.32026.03 | 37 | 83 | 81 | 72 | 70 | 79 | 9 | 57.25 | 88 | 0 | 75.67 | 82 | 50 | 60.3 | — | — | — | — | — | — | — | — | |
| EchoKV-HybridModel=Llama3.1-8B-Instruct, Compression Ratio=0.32026.03 | 31.4 | 57 | 79 | 46 | 88 | 78 | 17 | 71.25 | 93.5 | 0.1 | 67.67 | 82 | 50 | 58.53 | — | — | — | — | — | — | — | — | |
| PaluModel=Mistral-7B-Instruct-v0.3, Compression Ratio=0.32026.03 | 8.4 | 40 | 27 | 0 | 4 | 0 | 0 | 1.75 | 1 | 4.3 | 37.67 | 12 | 24 | 12.32 | — | — | — | — | — | — | — | — | |
| CommonKVModel=Mistral-7B-Instruct-v0.3, Compression Ratio=0.32026.03 | 1.6 | 11 | 0 | 0 | 2 | 0 | 0 | 0 | 1.25 | 12.3 | 27 | 15 | 25 | 7.32 | — | — | — | — | — | — | — | — | |
| ThinK*Model=Llama3.1-8B-Instruct, Compression Ratio=0.62026.03 | 0.6 | 1 | 2 | 0 | 2 | 0 | 0 | 1.75 | 3.25 | 0 | 48 | 63 | 40 | 12.43 | — | — | — | — | — | — | — | — | |
| PaluModel=Llama3.1-8B-Instruct, Compression Ratio=0.32026.03 | 0 | 11 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0.3 | 1 | 0 | 0 | 0.95 | — | — | — | — | — | — | — | — |