Lemma Judging on NaturalProofs (test)
93.2Exact Match AccuracyLlama-3.1 (RULES)
Evaluation results
| Method | Links | |
|---|---|---|
| Llama-3.1 (RULES)Model=Llama-3.1, Optimization=RULES, Evaluation=SC@72026.02 | 93.2 | |
| Llama-3.1 (GRPO)Model=Llama-3.1, Optimization=GRPO, Evaluation=SC@72026.02 | 88.6 | |
| DeepMath (RULES)Model=DeepMath, Optimization=RULES, Evaluation=SC@72026.02 | 88.5 | |
| Qwen2.5 (RULES)Model=Qwen2.5, Optimization=RULES, Evaluation=SC@72026.02 | 88.2 | |
| OLMO2 (GRPO)Model=OLMO2, Optimization=GRPO, Evaluation=SC@72026.02 | 85.2 | |
| OLMO2 (RULES)Model=OLMO2, Optimization=RULES, Evaluation=SC@72026.02 | 84.9 | |
| OLMO2 (vanilla)Model=OLMO2, Optimization=vanilla, Evaluation=SC@72026.02 | 77.6 | |
| DeepMath (GRPO)Model=DeepMath, Optimization=GRPO, Evaluation=SC@72026.02 | 77.5 | |
| Qwen2.5 (GRPO)Model=Qwen2.5, Optimization=GRPO, Evaluation=SC@72026.02 | 76.5 | |
| Qwen2.5 (vanilla)Model=Qwen2.5, Optimization=vanilla, Evaluation=SC@72026.02 | 63.9 | |
| Llama-3.1 (vanilla)Model=Llama-3.1, Optimization=vanilla, Evaluation=SC@72026.02 | 61.3 | |
| DeepMath (vanilla)Model=DeepMath, Optimization=vanilla, Evaluation=SC@72026.02 | 60.8 |