Commonsense Reasoning on OBQA, PIQA, ARC-e, ARC-c, HellaSwag, WinoGrande
44.2OBQA AccuracyLLaMA-2-7B
Evaluation results
| Method | Links | |||||||
|---|---|---|---|---|---|---|---|---|
| LLaMA-2-7BCompression Rate=0%2026.06 | 44.2 | 78.07 | 76.3 | 46.42 | 76 | 69.3 | 65.05 | |
| PaluBase Model=LLaMA-2-7B, Compression Rate=50%2026.06 | 43.6 | 76.33 | 73.02 | 42.57 | 73.39 | 66.67 | 62.6 | |
| STAR-KVBase Model=LLaMA-2-7B, Compression Rate=60%2026.06 | 43.4 | 79.22 | 75.55 | 44.71 | 74.63 | 68.51 | 64.34 | |
| STAR-KVBase Model=LLaMA-2-7B, Compression Rate=75%2026.06 | 43 | 78.18 | 73.78 | 41.64 | 73.14 | 66.61 | 62.73 | |
| LLaMA-3-8B-InstCompression Rate=0%2026.06 | 43 | 78.51 | 81.61 | 56.57 | 75.95 | 71.59 | 67.87 | |
| STAR-KVBase Model=LongChat-7B-v1.5, Compression Rate=75%2026.06 | 42.6 | 74.86 | 71.63 | 41.55 | 68.52 | 63.77 | 60.49 | |
| PaluBase Model=LLaMA-3-8B-Inst, Compression Rate=50%2026.06 | 42.4 | 76.12 | 76.14 | 49.06 | 70.33 | 71.82 | 64.31 | |
| STAR-KVBase Model=LLaMA-3-8B-Inst, Compression Rate=60%2026.06 | 42.2 | 78.13 | 79.29 | 49.83 | 73.37 | 69.69 | 65.42 | |
| STAR-KVBase Model=LLaMA-3-8B-Inst, Compression Rate=75%2026.06 | 42.2 | 78.02 | 77.57 | 49.32 | 71.59 | 66.22 | 64.15 | |
| STAR-KVBase Model=LongChat-7B-v1.5, Compression Rate=60%2026.06 | 41.8 | 75.9 | 72.69 | 41.47 | 70.49 | 66.85 | 61.53 | |
| LongChat-7B-v1.5Compression Rate=0%2026.06 | 41 | 76.28 | 71.84 | 41.38 | 71.2 | 67.48 | 61.53 | |
| ReCalKVBase Model=LLaMA-2-7B, Compression Rate=70%2026.06 | 39.8 | 74.48 | 70.37 | 39.42 | 69.59 | 65.75 | 59.9 | |
| PaluBase Model=LongChat-7B-v1.5, Compression Rate=50%2026.06 | 38.2 | 73.78 | 67.63 | 37.63 | 68.43 | 65.27 | 58.49 | |
| ReCalKVBase Model=LongChat-7B-v1.5, Compression Rate=70%2026.06 | 35.2 | 68.55 | 58.84 | 33.53 | 63.18 | 59.12 | 53.07 |