Zero-shot Question Answering on Reasoning Suite (OBQA, PIQA, ARC-e, ARC-c, Hella, Wino)
46.8Accuracy on OBQA (Zero-shot)Baseline
Evaluation results
| Method | Links | |||||||
|---|---|---|---|---|---|---|---|---|
| BaselineModel=Mistral-7B-Inst., Compression (%)=02026.06 | 46.8 | 80.41 | 81.31 | 55.63 | 83.48 | 74.35 | 70.33 | |
| BaselineModel=LLaMA-2-13B, Compression (%)=02026.06 | 44.2 | 79.22 | 77.57 | 50.51 | 79.71 | 71.03 | 67.04 | |
| ReCalKVModel=Mistral-7B-Inst., Compression (%)=602026.06 | 44 | 79.27 | 77.78 | 52.2 | 77.88 | 72.3 | 67.24 | |
| STAR-KVModel=Mistral-7B-Inst., Compression (%)=602026.06 | 44 | 80.52 | 79.8 | 52.13 | 78.79 | 71.11 | 67.73 | |
| STAR-KVModel=LLaMA-2-13B, Compression (%)=602026.06 | 43.6 | 79 | 77.36 | 48.12 | 79.3 | 71.27 | 66.44 | |
| PaluModel=Mistral-7B-Inst., Compression (%)=602026.06 | 42.8 | 77.26 | 74.07 | 50 | 75.3 | 70.72 | 65.03 | |
| STAR-KVModel=LLaMA-2-13B, Compression (%)=752026.06 | 42.4 | 77.97 | 75.8 | 46.16 | 77.79 | 68.35 | 64.75 | |
| PaluModel=LLaMA-2-13B, Compression (%)=602026.06 | 42 | 76.28 | 71.8 | 41.72 | 72.99 | 69.93 | 62.45 |