# Reasoning Trajectory Generation on Socratic Debugging of Student Code 227 samples

Source: [Reasoning Trajectory Generation on Socratic Debugging of Student Code 227 samples](<https://www.sota2.com/research/sota/reasoning-trajectory-generation-on-socratic-debugging-of-student-code-227-samples>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2026-05-12T13:35:34.473Z

Evaluation of model performance on generating reasoning trajectories for debugging student code, based on the Socratic Debugging of Student Code 227 samples dataset. Metrics include the number of reasoning steps and the validity rate of the generated trajectories.

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

| Scope | Recorded value |
| --- | --- |
| Task | Reasoning Trajectory Generation |
| Dataset (including any stated split) | Socratic Debugging of Student Code 227 samples |
| Metric | Reasoning Steps Count |
| Unit | SCALAR |
| Direction | Higher is better |
| Results loaded | 14 of 14 |
| Coverage | All results currently returned for this benchmark |

Evaluation settings may differ between results. The original papers and recorded configurations are needed to establish comparability. A shared metric name alone does not establish an identical protocol.

## Metrics

- [Reasoning Steps Count](<https://www.sota2.com/research/sota/reasoning-trajectory-generation-on-socratic-debugging-of-student-code-227-samples.md?metric=Reasoning%20Steps%20Count>): Higher is better; unit: SCALAR.

- [Validity Rate](<https://www.sota2.com/research/sota/reasoning-trajectory-generation-on-socratic-debugging-of-student-code-227-samples.md?metric=Validity%20Rate>): Higher is better; unit: PERCENT.

## Recorded results

| Method | Reasoning Steps Count | Result date | Report paper |
| --- | --- | --- | --- |
| Claude Haiku-4.5 | 1962 | Not recorded | 6909887b3ef7ccd74e5e5b24 |
| GPT-5-mini | 1826 | Not recorded | 6909887b3ef7ccd74e5e5b24 |
| Gemini 2.5-flash | 1826 | Not recorded | 6909887b3ef7ccd74e5e5b24 |
| Claude Sonnet-4.5 | 1792 | Not recorded | 6909887b3ef7ccd74e5e5b24 |
| Claude Sonnet-4.5 | 1776 | Not recorded | 6909887b3ef7ccd74e5e5b24 |
| Claude Haiku-4.5 | 1738 | Not recorded | 6909887b3ef7ccd74e5e5b24 |
| Gemini 2.5-pro | 1628 | Not recorded | 6909887b3ef7ccd74e5e5b24 |
| GPT-5 | 1577 | Not recorded | 6909887b3ef7ccd74e5e5b24 |
| GPT-5 | 1488 | Not recorded | 6909887b3ef7ccd74e5e5b24 |
| GPT-5-mini | 1453 | Not recorded | 6909887b3ef7ccd74e5e5b24 |
| Gemini 2.5-pro | 1439 | Not recorded | 6909887b3ef7ccd74e5e5b24 |
| Gemini 2.5-flash | 1379 | Not recorded | 6909887b3ef7ccd74e5e5b24 |
| GPT-5-mini | 1351 | Not recorded | 6909887b3ef7ccd74e5e5b24 |
| GPT-5 | 1271 | Not recorded | 6909887b3ef7ccd74e5e5b24 |

### Claude Haiku-4.5

Reasoning Steps Count: 1962. Result ID: 6a032ca247f34ce924e7afb8.

[Reporting paper](<https://www.sota2.com/research/paper/6909887b3ef7ccd74e5e5b24.md>)

Recorded evaluation configuration: {"Reasoning":"false"}. This record does not assert independent reproduction.

### GPT-5-mini

Reasoning Steps Count: 1826. Result ID: 6a032ca247f34ce924e7afb3.

[Reporting paper](<https://www.sota2.com/research/paper/6909887b3ef7ccd74e5e5b24.md>)

Recorded evaluation configuration: {"Effort level":"minimal-effort","Reasoning":"true"}. This record does not assert independent reproduction.

### Gemini 2.5-flash

Reasoning Steps Count: 1826. Result ID: 6a032ca247f34ce924e7afbb.

[Reporting paper](<https://www.sota2.com/research/paper/6909887b3ef7ccd74e5e5b24.md>)

Recorded evaluation configuration: {"Reasoning":"true"}. This record does not assert independent reproduction.

### Claude Sonnet-4.5

Reasoning Steps Count: 1792. Result ID: 6a032ca247f34ce924e7afb6.

[Reporting paper](<https://www.sota2.com/research/paper/6909887b3ef7ccd74e5e5b24.md>)

Recorded evaluation configuration: {"Reasoning":"false"}. This record does not assert independent reproduction.

### Claude Sonnet-4.5

Reasoning Steps Count: 1776. Result ID: 6a032ca247f34ce924e7afb7.

[Reporting paper](<https://www.sota2.com/research/paper/6909887b3ef7ccd74e5e5b24.md>)

Recorded evaluation configuration: {"Reasoning":"true"}. This record does not assert independent reproduction.

### Claude Haiku-4.5

Reasoning Steps Count: 1738. Result ID: 6a032ca247f34ce924e7afb9.

[Reporting paper](<https://www.sota2.com/research/paper/6909887b3ef7ccd74e5e5b24.md>)

Recorded evaluation configuration: {"Reasoning":"true"}. This record does not assert independent reproduction.

### Gemini 2.5-pro

Reasoning Steps Count: 1628. Result ID: 6a032ca247f34ce924e7afbd.

[Reporting paper](<https://www.sota2.com/research/paper/6909887b3ef7ccd74e5e5b24.md>)

Recorded evaluation configuration: {"Reasoning":"true"}. This record does not assert independent reproduction.

### GPT-5

Reasoning Steps Count: 1577. Result ID: 6a032ca247f34ce924e7afb0.

[Reporting paper](<https://www.sota2.com/research/paper/6909887b3ef7ccd74e5e5b24.md>)

Recorded evaluation configuration: {"Effort level":"minimal-effort","Reasoning":"true"}. This record does not assert independent reproduction.

### GPT-5

Reasoning Steps Count: 1488. Result ID: 6a032ca247f34ce924e7afb1.

[Reporting paper](<https://www.sota2.com/research/paper/6909887b3ef7ccd74e5e5b24.md>)

Recorded evaluation configuration: {"Effort level":"low-effort","Reasoning":"true"}. This record does not assert independent reproduction.

### GPT-5-mini

Reasoning Steps Count: 1453. Result ID: 6a032ca247f34ce924e7afb4.

[Reporting paper](<https://www.sota2.com/research/paper/6909887b3ef7ccd74e5e5b24.md>)

Recorded evaluation configuration: {"Effort level":"low-effort","Reasoning":"true"}. This record does not assert independent reproduction.

### Gemini 2.5-pro

Reasoning Steps Count: 1439. Result ID: 6a032ca247f34ce924e7afbc.

[Reporting paper](<https://www.sota2.com/research/paper/6909887b3ef7ccd74e5e5b24.md>)

Recorded evaluation configuration: {"Reasoning":"false"}. This record does not assert independent reproduction.

### Gemini 2.5-flash

Reasoning Steps Count: 1379. Result ID: 6a032ca247f34ce924e7afba.

[Reporting paper](<https://www.sota2.com/research/paper/6909887b3ef7ccd74e5e5b24.md>)

Recorded evaluation configuration: {"Reasoning":"false"}. This record does not assert independent reproduction.

### GPT-5-mini

Reasoning Steps Count: 1351. Result ID: 6a032ca247f34ce924e7afb5.

[Reporting paper](<https://www.sota2.com/research/paper/6909887b3ef7ccd74e5e5b24.md>)

Recorded evaluation configuration: {"Effort level":"medium-effort","Reasoning":"true"}. This record does not assert independent reproduction.

### GPT-5

Reasoning Steps Count: 1271. Result ID: 6a032ca247f34ce924e7afb2.

[Reporting paper](<https://www.sota2.com/research/paper/6909887b3ef7ccd74e5e5b24.md>)

Recorded evaluation configuration: {"Effort level":"medium-effort","Reasoning":"true"}. This record does not assert independent reproduction.

Page 1 of 1. This is an index page, not the full collection.
