Multiple-choice Question Answering and Reasoning on ARC-C, ARC-E, HellaSwag, PIQA, WinoGrande (test val)
22.7ARC-Challenge AccuracyQ-MQA
Evaluation results
| Method | Links | ||||||
|---|---|---|---|---|---|---|---|
| Q-MQAModel scale=1.2B, Evaluation protocol=5-shot, Cache reduction=96.9%2026.06 | 22.7 | 25.08 | 25.04 | 49.51 | 49.57 | 34.38 | |
| Q-GQA-8Model scale=1.2B, Evaluation protocol=5-shot, Cache reduction=87.5%2026.06 | 19.97 | 29.97 | 26.13 | 56.47 | 51.07 | 36.72 | |
| MQAModel scale=1.2B, Evaluation protocol=5-shot, Cache reduction=93.8%2026.06 | 19.62 | 30.09 | 26.23 | 55.77 | 50.12 | 36.37 | |
| QKVModel scale=1.2B, Evaluation protocol=5-shot, Cache reduction=—2026.06 | 19.03 | 30.01 | 26.15 | 56.64 | 50.2 | 36.4 | |
| Q-K=VModel scale=1.2B, Evaluation protocol=5-shot, Cache reduction=50%2026.06 | 18.94 | 28.62 | 26.14 | 56.37 | 49.88 | 35.99 | |
| GQA-8Model scale=1.2B, Evaluation protocol=5-shot, Cache reduction=75%2026.06 | 18.69 | 29.42 | 26.44 | 56.91 | 47.83 | 35.86 |