Mathematical Reasoning on GSM8K (PV, Best, Alpha Metrics)
97.7Ac (alpha=0.10)Oracle
Evaluation results
| Method | Links | ||||||
|---|---|---|---|---|---|---|---|
| OracleModel=Gemma-4-E4B, n_cal=200, m=16, splits=202026.05 | 97.7 | — | 97.7 | 100 | 97.7 | 100 | |
| OracleModel=R1-Qwen-1.5B, n_cal=200, m=16, splits=202026.05 | 95.6 | — | 95.6 | 100 | 95.6 | 100 | |
| R-Q8BModel=Gemma-4-E4B, n_cal=200, m=16, splits=202026.05 | 92.4 | 91.4 | 84.8 | 98.5 | 95.2 | 89.5 | |
| MVModel=Gemma-4-E4B, n_cal=200, m=16, splits=202026.05 | 91.4 | 90.1 | — | 97.2 | 95.2 | 79.2 | |
| R-L8BModel=Gemma-4-E4B, n_cal=200, m=16, splits=202026.05 | 91.4 | 90.7 | 83.8 | 98.7 | 95.2 | 85.2 | |
| J-Q30BModel=Gemma-4-E4B, n_cal=200, m=16, splits=202026.05 | 91.3 | 90.2 | 68.8 | 97.4 | 94.8 | 81.1 | |
| Ppl-StdModel=Gemma-4-E4B, n_cal=200, m=16, splits=202026.05 | 91 | 88.8 | 76 | 94.1 | 94.5 | 77.4 | |
| R-Q8BModel=R1-Qwen-1.5B, n_cal=200, m=16, splits=202026.05 | 90.9 | 88.3 | 88.3 | 95.4 | 95.1 | 81.8 | |
| J-G26BModel=Gemma-4-E4B, n_cal=200, m=16, splits=202026.05 | 90.9 | 88.7 | 67.5 | 95.2 | 94.9 | 77.4 | |
| PplModel=Gemma-4-E4B, n_cal=200, m=16, splits=202026.05 | 90.5 | 88.2 | 78 | 94.7 | 94.5 | 80.5 | |
| R-L8BModel=R1-Qwen-1.5B, n_cal=200, m=16, splits=202026.05 | 90.4 | 87.3 | 84.2 | 94.4 | 95 | 79.7 | |
| OracleModel=Qwen3-1.7B, n_cal=200, m=16, splits=202026.05 | 89.9 | — | 89.9 | 100 | 89.9 | 100 | |
| J-G26BModel=R1-Qwen-1.5B, n_cal=200, m=16, splits=202026.05 | 89.7 | 85.2 | 82.9 | 92 | 94.3 | 75 | |
| MVModel=R1-Qwen-1.5B, n_cal=200, m=16, splits=202026.05 | 89.6 | 82.2 | — | 84 | 94.2 | 64.7 | |
| PplModel=R1-Qwen-1.5B, n_cal=200, m=16, splits=202026.05 | 89.6 | 82.6 | 77.9 | 87.3 | 94.4 | 68.8 | |
| J-R1Q14BModel=Gemma-4-E4B, n_cal=200, m=16, splits=202026.05 | 89.6 | 84.7 | 64.1 | 88.8 | 94 | 71.2 | |
| J-Q30BModel=R1-Qwen-1.5B, n_cal=200, m=16, splits=202026.05 | 89.4 | 83.8 | 70.3 | 88.2 | 94.4 | 68.7 | |
| Ppl-StdModel=R1-Qwen-1.5B, n_cal=200, m=16, splits=202026.05 | 89.4 | 81.5 | 72.5 | 84.2 | 93.9 | 65.9 | |
| J-R1Q14BModel=R1-Qwen-1.5B, n_cal=200, m=16, splits=202026.05 | 89.3 | 82.8 | 78.2 | 86.4 | 94.1 | 69.5 | |
| MVModel=Qwen3-1.7B, n_cal=200, m=16, splits=202026.05 | 88.8 | 73 | — | 71.3 | 92.9 | 61.7 | |
| R-Q8BModel=Qwen3-1.7B, n_cal=200, m=16, splits=202026.05 | 87.7 | 75.7 | 69.7 | 77.7 | 92.9 | 64.5 | |
| J-Q30BModel=Qwen3-1.7B, n_cal=200, m=16, splits=202026.05 | 87.7 | 73.6 | 59.1 | 73.5 | 92.4 | 62.5 | |
| R-L8BModel=Qwen3-1.7B, n_cal=200, m=16, splits=202026.05 | 87.2 | 74 | 65.5 | 74.7 | 92.8 | 62.9 | |
| J-R1Q14BModel=Qwen3-1.7B, n_cal=200, m=16, splits=202026.05 | 87.2 | 73.7 | 61.2 | 74.8 | 92.3 | 62.2 | |
| J-G26BModel=Qwen3-1.7B, n_cal=200, m=16, splits=202026.05 | 86.9 | 72.3 | 59.9 | 74.1 | 92.5 | 62.1 | |
| PplModel=Qwen3-1.7B, n_cal=200, m=16, splits=202026.05 | 86.6 | 73.3 | 64.9 | 75.8 | 92.3 | 61.7 | |
| Ppl-StdModel=Qwen3-1.7B, n_cal=200, m=16, splits=202026.05 | 86.1 | 69.8 | 57.4 | 70.2 | 91.2 | 56.1 | |
| GreedyModel=Gemma-4-E4B, n_cal=200, m=16, splits=202026.05 | 68 | — | — | 100 | 68 | 100 | |
| GreedyModel=R1-Qwen-1.5B, n_cal=200, m=16, splits=202026.05 | 62.4 | — | — | 100 | 62.4 | 100 | |
| GreedyModel=Qwen3-1.7B, n_cal=200, m=16, splits=202026.05 | 58.3 | — | — | 100 | 58.3 | 100 |