Tool-augmented Reasoning on PYMATH (test)
71.9Final AccuracyGPT-5-Thinking
Evaluation Results
| Method | Links | ||||
|---|---|---|---|---|---|
| GPT-5-ThinkingVariant=TaLM2025.11 | 71.9 | 43.8 | 44 | 50.2 | |
| GPT-5-ThinkingVariant=Base2025.11 | 67.5 | 38.8 | 56 | 57.5 | |
| o4-miniVariant=TaLM2025.11 | 64.4 | 47.6 | 50.9 | 67.9 | |
| Gemini-2.5-FlashVariant=TaLM2025.11 | 45.7 | 40.9 | 45.4 | 78.8 | |
| Gemini-2.5-FlashVariant=Base2025.11 | 45.4 | 40.2 | 54.6 | 81.5 | |
| o4-miniVariant=Base2025.11 | 45.1 | 45.5 | 49 | 73.6 | |
| Claude-Opus-4Variant=TaLM2025.11 | 40.5 | 52.8 | 58.6 | 57.8 | |
| GPT-4.1-miniVariant=Base2025.11 | 30 | 45.7 | 58.6 | 93 | |
| GPT-4.1-miniVariant=TaLM2025.11 | 28.7 | 49.9 | 41.4 | 88.9 | |
| Claude-Opus-4Variant=Base2025.11 | 28 | 50.9 | 41.3 | 77.9 | |
| GPT-4.1Variant=TaLM2025.11 | 27 | 49.9 | 45.6 | 85.9 | |
| Gemini-2.0-FlashVariant=TaLM2025.11 | 25.1 | 56.6 | 47.3 | 68.5 | |
| GPT-4.1Variant=Base2025.11 | 24.6 | 48.1 | 54.4 | 88.6 | |
| Gemini-2.0-FlashVariant=Base2025.11 | 24.3 | 54.2 | 52.7 | 65 |