Mathematical Reasoning on GSM8K (test) (Exact Match)
38.36Exact Match Accuracyfp16 (KIVI paper)
Evaluation results
| Method | Links | |
|---|---|---|
| fp16 (KIVI paper)bits=16.02026.05 | 38.36 | |
| KIVI-4 (calib.)bits=4.52026.05 | 37.3 | |
| KIVI-2 (calib.)bits=2.52026.05 | 36.01 | |
| fp16 (our run)bits=16.02026.05 | 35.5 | |
| HQMQ s96 r4bits=3.792026.05 | 35 | |
| HQMQ s96 r4 + Med3×bits=4.412026.05 | 34.5 | |
| HQMQ s24 r2bits=2.792026.05 | 8.5 | |
| naive int2bits=2.002026.05 | 0 |