Reasoning Trajectory Generation on Socratic Debugging of Student Code 227 samples
1,962Reasoning Steps CountClaude Haiku-4.5
Evaluation Results
| Method | Links | ||
|---|---|---|---|
| Claude Haiku-4.5Reasoning=false2025.11 | 1,962 | 62.6 | |
| GPT-5-miniEffort level=minimal-effort, Reasoning=true2025.11 | 1,826 | 68.3 | |
| Gemini 2.5-flashReasoning=true2025.11 | 1,826 | 82.7 | |
| Claude Sonnet-4.5Reasoning=false2025.11 | 1,792 | 80.6 | |
| Claude Sonnet-4.5Reasoning=true2025.11 | 1,776 | 87.2 | |
| Claude Haiku-4.5Reasoning=true2025.11 | 1,738 | 78.9 | |
| Gemini 2.5-proReasoning=true2025.11 | 1,628 | 85.3 | |
| GPT-5Effort level=minimal-effort, Reasoning=true2025.11 | 1,577 | 85 | |
| GPT-5Effort level=low-effort, Reasoning=true2025.11 | 1,488 | 90.7 | |
| GPT-5-miniEffort level=low-effort, Reasoning=true2025.11 | 1,453 | 59.5 | |
| Gemini 2.5-proReasoning=false2025.11 | 1,439 | 77.2 | |
| Gemini 2.5-flashReasoning=false2025.11 | 1,379 | 83.5 | |
| GPT-5-miniEffort level=medium-effort, Reasoning=true2025.11 | 1,351 | 68.9 | |
| GPT-5Effort level=medium-effort, Reasoning=true2025.11 | 1,271 | 91.1 |