Long-context language understanding on LongBench (12 specific metrics subset)
30.54NrtvQAPalu
Evaluation results
| Method | Links | ||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| PaluLLM Backbone=Llama-3.1-8B, Compression Ratio=0.8x2025.09 | 30.54 | 32.91 | 21.38 | 18.07 | 14.76 | 71 | 4.07 | 3 | 93.14 | 5.14 | 5.86 | 27.26 | |
| Full KVLLM Backbone=Llama-3.1-8B2025.09 | 29.56 | 53 | 53.76 | 46.13 | 28.38 | 7.5 | 7.47 | 6.25 | 99.5 | 19.88 | 19.22 | 33.7 | |
| PaluLLM Backbone=Llama-3.1-8B, Compression Ratio=0.6x2025.09 | 24.25 | 28.35 | 18.72 | 16.53 | 10.86 | 65 | 4.28 | 3 | 20.49 | 2.81 | 3.23 | 17.96 | |
| Full KVLLM Backbone=Llama-2-7B2025.09 | 18.79 | 34.41 | 25.3 | 28.33 | 8.52 | 0 | 6.22 | 1.55 | 9 | 15.08 | 17.35 | 14.96 | |
| PaluLLM Backbone=Llama-2-7B, Compression Ratio=0.6x2025.09 | 17.17 | 23.63 | 9.71 | 10.22 | 5.3 | 0 | 3.86 | 3.16 | 7.1 | 5.08 | 4.77 | 8.18 | |
| StaticPCALLM Backbone=Llama-2-7B, Compression Ratio=0.8x2025.09 | 16.95 | 32.8 | 21.31 | 24.73 | 6.28 | 0 | 5.34 | 2.25 | 2.61 | 14.06 | 16.78 | 13.01 | |
| OjaKV-PFLLM Backbone=Llama-2-7B, Compression Ratio=0.6x2025.09 | 16.86 | 33.76 | 23.49 | 25.91 | 7.42 | 0 | 5.26 | 3.14 | 3.75 | 14.65 | 16.79 | 13.73 | |
| OjaKV-PFLLM Backbone=Llama-2-7B, Compression Ratio=0.8x2025.09 | 16.77 | 33.62 | 23.3 | 26.38 | 7.33 | 0 | 5.28 | 4 | 1.5 | 16.05 | 17.48 | 13.79 | |
| OjaKVLLM Backbone=Llama-2-7B, Compression Ratio=0.8x2025.09 | 16.28 | 31.19 | 20.4 | 23.7 | 7.44 | 0 | 3.91 | 1.61 | 4 | 16.16 | 20.85 | 13.23 | |
| OjaKVLLM Backbone=Llama-2-7B, Compression Ratio=0.6x2025.09 | 15.91 | 31.48 | 19.97 | 20.34 | 7.69 | 0 | 5.76 | 2.01 | 2.28 | 16.06 | 21.15 | 12.97 | |
| PaluLLM Backbone=Llama-2-7B, Compression Ratio=0.8x2025.09 | 15.75 | 21.32 | 8.5 | 10.11 | 4.34 | 0 | 4.14 | 2.35 | 6.58 | 6.48 | 5.31 | 7.72 | |
| StaticPCALLM Backbone=Llama-2-7B, Compression Ratio=0.8x2025.09 | 14.21 | 27.54 | 17.41 | 19.77 | 5.11 | 0.25 | 5.65 | 3.01 | 0.5 | 13.41 | 18.84 | 11.43 | |
| OjaKVLLM Backbone=Llama-3.1-8B, Compression Ratio=0.8x2025.09 | 12.34 | 48.89 | 49.26 | 44.69 | 17.47 | 9 | 9.59 | 3 | 89 | 21.71 | 22.92 | 29.81 | |
| OjaKV-PFLLM Backbone=Llama-3.1-8B, Compression Ratio=0.8x2025.09 | 12.11 | 48.81 | 48.94 | 44.58 | 18.87 | 10 | 7.88 | 4 | 87.5 | 21.2 | 22.35 | 29.66 | |
| StaticPCALLM Backbone=Llama-3.1-8B, Compression Ratio=0.8x2025.09 | 11.81 | 49.68 | 48.06 | 43.56 | 15.43 | 10 | 9.46 | 3 | 89.5 | 21.36 | 22.46 | 29.48 | |
| OjaKV-PFLLM Backbone=Llama-3.1-8B, Compression Ratio=0.6x2025.09 | 11.63 | 47.99 | 46.98 | 43.71 | 18.87 | 4 | 7.88 | 4 | 87.5 | 21.59 | 22.97 | 28.46 | |
| OjaKVLLM Backbone=Llama-3.1-8B, Compression Ratio=0.6x2025.09 | 9.73 | 43.94 | 35.95 | 34.49 | 15.21 | 3.5 | 7.78 | 3.5 | 33 | 18.21 | 21.94 | 20.66 | |
| StaticPCALLM Backbone=Llama-3.1-8B, Compression Ratio=0.6x2025.09 | 8.16 | 39.38 | 25.63 | 22.73 | 12.65 | 0.67 | 7.1 | 1.62 | 12.5 | 18.76 | 23.48 | 15.7 |