Lemma Judging on IMO-Lemma (IL) (test)
93.6Exact Match AccuracyLlama-3.1 (RULES)
Evaluation results
| Method | Links | |
|---|---|---|
| Llama-3.1 (RULES)Optimization=RULES2026.02 | 93.6 | |
| Qwen2.5 (RULES)Optimization=RULES2026.02 | 92.7 | |
| Llama-3.1 (GRPO)Optimization=GRPO2026.02 | 89.8 | |
| DeepMath (RULES)Optimization=RULES2026.02 | 89.5 | |
| OLMO2 (GRPO)Optimization=GRPO2026.02 | 83.6 | |
| OLMO2 (RULES)Optimization=RULES2026.02 | 82.5 | |
| Qwen2.5 (GRPO)Optimization=GRPO2026.02 | 80.4 | |
| DeepMath (GRPO)Optimization=GRPO2026.02 | 65.8 | |
| OLMO2 (vanilla)Optimization=vanilla2026.02 | 64.9 | |
| Qwen2.5 (vanilla)Optimization=vanilla2026.02 | 55.8 | |
| Llama-3.1 (vanilla)Optimization=vanilla2026.02 | 52.3 | |
| DeepMath (vanilla)Optimization=vanilla2026.02 | 31.9 |