Mathematical Reasoning on MATH Subset
44.2AccuracySCORE (Oracle Verifier)
Evaluation results
| Method | Links | ||
|---|---|---|---|
| SCORE (Oracle Verifier)Base LM=Gemma-7B-it, Verifier=oracle, Refiner=SCORE (fine-tuned)2024.04 | 44.2 | 100 | |
| SCORE (GPT-4 Verifier)Base LM=Gemma-7B-it, Verifier=gpt-4, Refiner=SCORE (fine-tuned)2024.04 | 39.2 | 82.8 | |
| SCORE (Oracle Verifier)Base LM=LLaMA-2-13B-chat, Verifier=oracle, Refiner=SCORE (fine-tuned)2024.04 | 31.5 | 100 | |
| Oracle Verifier + Prompted RefinerBase LM=Gemma-7B-it, Verifier=oracle, Refiner=prompted2024.04 | 29.3 | 100 | |
| Prompted Verifier + Prompted RefinerBase LM=Gemma-7B-it, Verifier=prompted, Refiner=prompted2024.04 | 28.2 | 39.2 | |
| Few-shot promptingBase LM=Gemma-7B-it2024.04 | 27.1 | — | |
| SCORE (Self Verifier)Base LM=Gemma-7B-it, Verifier=self, Refiner=SCORE (fine-tuned)2024.04 | 27.1 | 49.5 | |
| Oracle Verifier + Prompted RefinerBase LM=LLaMA-2-13B-chat, Verifier=oracle, Refiner=prompted2024.04 | 26.5 | 100 | |
| SCORE (GPT-4 Verifier)Base LM=LLaMA-2-13B-chat, Verifier=gpt-4, Refiner=SCORE (fine-tuned)2024.04 | 26.5 | 82.9 | |
| Few-shot promptingBase LM=LLaMA-2-13B-chat, Decoding Strategy=greedy2024.04 | 23.8 | — | |
| Prompted Verifier + Prompted RefinerBase LM=LLaMA-2-13B-chat, Verifier=prompted, Refiner=prompted2024.04 | 23.8 | 20 | |
| SCORE (Self Verifier)Base LM=LLaMA-2-13B-chat, Verifier=self, Refiner=SCORE (fine-tuned)2024.04 | 23.8 | 36 |