Question Answering and Commonsense Reasoning on ARC-C, ARC-E, PIQA, Winogrande, HellaSwag
63.48ARC-C AccuracyFP16/BF16
Evaluation results
| Method | Links | ||||||
|---|---|---|---|---|---|---|---|
| FP16/BF16Backbone=Llama-3.1-70B-Instruct, bit/param=162026.04 | 63.48 | 83.92 | 83.95 | 79.01 | 84.58 | 78.99 | |
| QuIPBackbone=Llama-3.1-70B-Instruct, bit/param=3.252026.04 | 62.12 | 82.45 | 82.1 | 78.06 | 82.83 | 77.51 | |
| GSQBackbone=Llama-3.1-70B-Instruct, bit/param=3.132026.04 | 62.12 | 82.83 | 82.75 | 78.93 | 83.3 | 77.99 | |
| EfficientQATBackbone=Llama-3.1-70B-Instruct, bit/param=3.252026.04 | 61.86 | 83.88 | 82.48 | 76.01 | 82.79 | 77.4 | |
| QTIPBackbone=Llama-3.1-70B-Instruct, bit/param=3.002026.04 | 61.77 | 82.79 | 83.79 | 78.3 | 84.21 | 78.17 | |
| QTIPBackbone=Llama-3.1-70B-Instruct, bit/param=2.002026.04 | 61.69 | 81.69 | 82.43 | 77.51 | 82.95 | 77.25 | |
| GPTQBackbone=Llama-3.1-70B-Instruct, bit/param=3.252026.04 | 60.75 | 80.64 | 83.3 | 77.98 | 82.37 | 77.01 | |
| GSQBackbone=Llama-3.1-70B-Instruct, bit/param=2.37^NU2026.04 | 60.2 | 81.4 | 82.4 | 80.6 | 82.6 | 77.4 | |
| GSQBackbone=Llama-3.1-70B-Instruct, bit/param=2.62^NU2026.04 | 59.6 | 81.4 | 82.9 | 80.4 | 82.9 | 77.5 | |
| GSQBackbone=Llama-3.1-70B-Instruct, bit/param=2.132026.04 | 58.87 | 79.55 | 81.07 | 76.24 | 82.11 | 75.57 | |
| PV-TuningBackbone=Llama-3.1-70B-Instruct, bit/param=2.072026.04 | 58.62 | 80.72 | 81.56 | 77.74 | 82.72 | 76.27 | |
| FP16/BF16Backbone=Llama-3.1-8B-Instruct, bit/param=162026.04 | 55.12 | 79.63 | 80.85 | 73.8 | 79.16 | 73.71 | |
| EfficientQATBackbone=Llama-3.1-70B-Instruct, bit/param=2.372026.04 | 54.86 | 77.27 | 80.36 | 65.67 | 79.01 | 71.43 | |
| QTIPBackbone=Llama-3.1-8B-Instruct, bit/param=3.002026.04 | 53.92 | 79.42 | 80.25 | 72.14 | 78.3 | 72.81 | |
| EfficientQATBackbone=Llama-3.1-8B-Instruct, bit/param=3.252026.04 | 52.99 | 78.91 | 79.76 | 71.59 | 76.85 | 72.02 | |
| GSQBackbone=Llama-3.1-8B-Instruct, bit/param=3.132026.04 | 52.99 | 78.37 | 80.03 | 73.56 | 76.66 | 72.32 | |
| QuIPBackbone=Llama-3.1-8B-Instruct, bit/param=3.252026.04 | 52.3 | 76.68 | 78.84 | 71.9 | 75.37 | 71.02 | |
| QTIPBackbone=Llama-3.1-8B-Instruct, bit/param=2.002026.04 | 50.68 | 75.42 | 78.18 | 70.09 | 75.02 | 69.88 | |
| GSQBackbone=Llama-3.1-8B-Instruct, bit/param=2.37^NU2026.04 | 50.26 | 74.41 | 78.13 | 69.61 | 74.81 | 69.44 | |
| PV-TuningBackbone=Llama-3.1-8B-Instruct, bit/param=2.272026.04 | 50.26 | 73.91 | 79.16 | 70.56 | 75.28 | 69.83 | |
| GSQBackbone=Llama-3.1-8B-Instruct, bit/param=2.132026.04 | 48.12 | 72.35 | 78.07 | 70.8 | 73.42 | 68.55 | |
| Full CacheBackbone=Llama-3-8B, Rate=100%, Evaluation Protocol=Zero-shot2026.02 | 45.76 | 70.02 | 80.85 | 71.51 | 74.68 | 68.56 | |
| DynaKVBackbone=Llama-3-8B, Rate=47%, Evaluation Protocol=Zero-shot2026.02 | 44.75 | 67.37 | 80.36 | 67.64 | 71.7 | 66.36 | |
| Full CacheBackbone=Qwen2-7B-Base, Rate=100%, Evaluation Protocol=Zero-shot2026.02 | 44.41 | 71.6 | 79.76 | 70.4 | 72.28 | 67.69 | |
| EfficientQATBackbone=Llama-3.1-8B-Instruct, bit/param=2.372026.04 | 43.77 | 67.55 | 74.65 | 64.33 | 68.65 | 63.79 | |
| GSQBackbone=Llama-3.1-8B-Instruct, bit/param=1.712026.04 | 42.83 | 67.13 | 73.5 | 65.82 | 67.91 | 63.44 | |
| DynaKVBackbone=Llama-3-8B, Rate=29%, Evaluation Protocol=Zero-shot2026.02 | 42.71 | 59.96 | 79.82 | 66.54 | 70.33 | 63.87 | |
| DynaKVBackbone=Qwen2-7B-Base, Rate=48%, Evaluation Protocol=Zero-shot2026.02 | 42.37 | 70.19 | 78.78 | 64.96 | 68.18 | 64.9 | |
| DynaKVBackbone=Qwen2-7B-Base, Rate=30%, Evaluation Protocol=Zero-shot2026.02 | 40 | 50.79 | 76.71 | 63.14 | 67.08 | 59.54 | |
| GPTQBackbone=Llama-3.1-8B-Instruct, bit/param=3.252026.04 | 39.68 | 58.67 | 67.03 | 66.3 | 64.24 | 59.18 | |
| QuIPBackbone=Llama-3.1-70B-Instruct, bit/param=2.372026.04 | 39.51 | 60.44 | 73.61 | 65.35 | 68.95 | 61.57 | |
| PaluBackbone=Llama-3-8B, Rate=50%, Evaluation Protocol=Zero-shot2026.02 | 39.32 | 63.84 | 77.58 | 67.8 | 69 | 63.51 | |
| DynaKVBackbone=Qwen2-7B-Base, Rate=18%, Evaluation Protocol=Zero-shot2026.02 | 39.32 | 59.96 | 75.41 | 58.72 | 62.44 | 59.17 | |
| DynaKVBackbone=Llama-3-8B, Rate=20%, Evaluation Protocol=Zero-shot2026.02 | 38.98 | 59.44 | 78.56 | 65.51 | 67.89 | 62.08 | |
| MatryoshkaKVBackbone=Llama-3-8B, Rate=50%, Evaluation Protocol=Zero-shot2026.02 | 38.64 | 60.49 | 78.02 | 63.14 | 67.36 | 61.53 | |
| GPTQBackbone=Llama-3.1-70B-Instruct, bit/param=2.372026.04 | 38.23 | 58.63 | 72.8 | 57.14 | 60.11 | 57.38 | |
| MatryoshkaKVBackbone=Llama-3-8B, Rate=30%, Evaluation Protocol=Zero-shot2026.02 | 32.88 | 49.74 | 73.67 | 55.8 | 56.8 | 53.78 | |
| PaluBackbone=Qwen2-7B-Base, Rate=50%, Evaluation Protocol=Zero-shot2026.02 | 32.88 | 49.74 | 71.38 | 62.43 | 60.74 | 55.43 | |
| PaluBackbone=Llama-3-8B, Rate=20%, Evaluation Protocol=Zero-shot2026.02 | 29.49 | 28.57 | 65.72 | 54.7 | 46.47 | 44.99 | |
| PaluBackbone=Llama-3-8B, Rate=30%, Evaluation Protocol=Zero-shot2026.02 | 29.15 | 27.51 | 71.6 | 60.62 | 57.19 | 49.21 | |
| MatryoshkaKVBackbone=Llama-3-8B, Rate=20%, Evaluation Protocol=Zero-shot2026.02 | 27.46 | 45.15 | 68.5 | 53.83 | 45.31 | 48.05 | |
| PaluBackbone=Qwen2-7B-Base, Rate=30%, Evaluation Protocol=Zero-shot2026.02 | 27.12 | 37.74 | 61.97 | 53.75 | 40.04 | 44.12 | |
| PaluBackbone=Qwen2-7B-Base, Rate=20%, Evaluation Protocol=Zero-shot2026.02 | 26.44 | 31.57 | 56.91 | 50.99 | 32.39 | 39.66 | |
| GPTQBackbone=Llama-3.1-8B-Instruct, bit/param=2.372026.04 | 24.91 | 27.61 | 52.83 | 51.78 | 30.5 | 37.53 | |
| QuIPBackbone=Llama-3.1-8B-Instruct, bit/param=2.372026.04 | 23.72 | 28.96 | 53.86 | 50.83 | 38.65 | 39.2 |