Mathematical Reasoning on MATH 500 (Acc., TPS)
74.8AccuracySpec (K=3)
Evaluation Results
| Method | Links | ||
|---|---|---|---|
| Spec (K=3)Model scale=8B, K=32026.05 | 74.8 | 1.35 | |
| Base (r=1)Model scale=8B, r=12026.05 | 72.2 | 114 | |
| Base (r=1)Model scale=4B, r=12026.05 | 71.4 | 165.4 | |
| Direct (K=1)Model scale=8B, K=12026.05 | 71.4 | 1.6 | |
| Direct (K=2)Model scale=8B, K=22026.05 | 70.8 | 1.84 | |
| Base (r=2)Model scale=8B, r=22026.05 | 68.4 | 1.04 | |
| Spec (K=3)Model scale=4B, K=32026.05 | 68 | 1.3 | |
| Direct (K=1)Model scale=4B, K=12026.05 | 64.4 | 1.6 | |
| Base (r=2)Model scale=4B, r=22026.05 | 64 | 1.02 | |
| Direct (K=2)Model scale=4B, K=22026.05 | 60.4 | 1.79 | |
| Spec (K=3)Model scale=1.7B, K=32026.05 | 58.4 | 1.23 | |
| Base (r=1)Model scale=1.7B, r=12026.05 | 55 | 277.6 | |
| Direct (K=1)Model scale=1.7B, K=12026.05 | 54.2 | 1.5 | |
| Direct (K=2)Model scale=1.7B, K=22026.05 | 50.6 | 1.67 | |
| Base (r=2)Model scale=1.7B, r=22026.05 | 45.8 | 0.92 | |
| FP16Backbone=LLaMA-3-8B-Instruct, BPA=162026.07 | 20.4 | — | |
| GSRQBackbone=LLaMA-3-8B-Instruct, BPA=22026.07 | 19 | — | |
| KIVIBackbone=LLaMA-3-8B-Instruct, BPA=22026.07 | 15.8 | — | |
| GSRQBackbone=LLaMA-3-8B-Instruct, BPA=12026.07 | 14.2 | — | |
| VQLLMBackbone=LLaMA-3-8B-Instruct, BPA=22026.07 | 13.4 | — | |
| KIVIBackbone=LLaMA-3-8B-Instruct, BPA=12026.07 | 0.8 | — | |
| VQLLMBackbone=LLaMA-3-8B-Instruct, BPA=12026.07 | 0.2 | — |