Code trace prediction on TraceEval stack-edu (held-out)
5.6Exact MatchClaude 4.5 Sonnet (tools off)
Evaluation results
| Method | Links | ||
|---|---|---|---|
| Claude 4.5 Sonnet (tools off)tools=off2026.06 | 5.6 | 30.8 | |
| max(gpt-oss-20b medium, gpt-oss-20b high)aggregation=max2026.06 | 1.9 | 19.9 | |
| gpt-oss-20b (medium)reasoning_length=medium2026.06 | 1.4 | 17 | |
| gpt-oss-20b (high)reasoning_length=high2026.06 | 1 | 8.9 | |
| gemma-3-27b-itvariant=it2026.06 | 0.4 | 9.5 |