# Playing Psychic: Using Thought Trees to Predict Reasoning Models Accuracy on Coding Tasks Source: [Playing Psychic: Using Thought Trees to Predict Reasoning Models Accuracy on Coding Tasks]() Collection index: [llms.txt]() Data updated: 2026-04-22T05:21:28.943Z Authors: Jiaxin Fang, Runyuan He, Sahil Bhatia, Neel Gajare, Alvin Cheung Published: 2026-04-18T09:30:36.000Z ## Abstract Recent advances in large language models (LLMs) have shown that test-time scaling can substantially improve model performance on complex tasks, particularly in the coding domain. Under this paradigm, models use a larger token budget during inference to generate intermediate reasoning traces before producing a final answer. However, current evaluations primarily rely on competitive programming benchmarks, which may not capture the full range of reasoning abilities. In this work, we perform a systematic study of frontier reasoning models to understand their performance on real-world coding benchmarks. To gain more insights into the performance of such models, we devise a programmatic way to {\\em automatically generate} coding tasks of arbitrary difficulty and structure from existing benchmarks. Using this framework, our analysis reveals that the structure of a reasoning trace, not just its contents, is a strong predictor of correctness. Motivated by this, we propose structured thought-trees as means to represent reasoning traces. To illustrate their use, we train a lightweight classifier on features extracted from thought-trees to predict trace correctness, and demonstrate that flagging and retrying structurally anomalous traces based on the extracted features yields consistent gains at lower complexity levels. ## Recorded benchmark results A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage. - [Reasoning failure prediction and recovery on CRUXEval L2](): Reasoning failure prediction and recovery; CRUXEval L2; Accuracy: 77. - [Reasoning failure prediction on CodeLingua (L1)](): Reasoning failure prediction; CodeLingua (L1); Accuracy: 73. - [Reasoning failure prediction on CodeLingua (L2)](): Reasoning failure prediction; CodeLingua (L2); Accuracy: 75. - [Reasoning failure prediction on CodeLingua (L3)](): Reasoning failure prediction; CodeLingua (L3); Accuracy: 76. - [Reasoning failure prediction and recovery on CRUXEval L1](): Reasoning failure prediction and recovery; CRUXEval L1; Accuracy: 89. - [Reasoning failure prediction and recovery on CRUXEval (L3)](): Reasoning failure prediction and recovery; CRUXEval (L3); Accuracy: 74. - [Reasoning failure prediction and recovery on SAFIM L1](): Reasoning failure prediction and recovery; SAFIM L1; Accuracy: 79. - [Reasoning failure prediction and recovery on SAFIM L3](): Reasoning failure prediction and recovery; SAFIM L3; Accuracy: 81. - [Block Infilling on SAFIM](): Block Infilling; SAFIM; Pass@1: not recorded. - [Code Translation on CodeLingua Java → C](): Code Translation; CodeLingua Java → C; Pass@1: not recorded. - [Code Translation on CodeLingua Python → C](): Code Translation; CodeLingua Python → C; Pass@1 Accuracy: not recorded. - [Code Translation on CodeLingua Java → Fortran](): Code Translation; CodeLingua Java → Fortran; Pass@1: not recorded. - [Output Prediction on CruxEval](): Output Prediction; CRUXEval; Pass@1: not recorded. ## Original sources and implementation [Original paper]() [PDF]()