# Measuring Judgment Quality in Natural-Language Explanations: Evidence from Forecasting Tournaments Source: [Measuring Judgment Quality in Natural-Language Explanations: Evidence from Forecasting Tournaments]() Collection index: [llms.txt]() Data updated: 2026-07-02T05:00:05.660Z Authors: Christopher W. Karvetski, Sheldon S. Huang, Simas Ku\\v{c}inskas, Nadja Flechner, Jingyu Hu, Philip Tetlock, Ezra Karger Published: 2026-06-29T23:51:09.000Z ## Abstract Decision-makers routinely rely on expert judgments accompanied by written explanations, yet explanation quality is difficult to measure at scale. Forecasting tournaments offer a natural testing ground: probabilistic judgments are paired with natural-language rationales and scored against realized outcomes. We introduce Explanation Quality Markers (EQMs), a set of sixty theory-guided reasoning patterns scored by large language models (LLMs). In a pre-registered analysis of over 55,000 forecast-rationale pairs from a multiyear forecasting tournament, EQMs predict accuracy at both the forecast and forecaster levels, consistently outperforming pre-LLM text-analysis methods. More than 90% of statistically significant pattern-level EQM-accuracy correlations match our directional hypotheses. The signal is asymmetric: EQMs identify likely underperformers more reliably than they distinguish the very best forecasters. Benchmarked against traditional indicators of forecasting skill, EQMs are the strongest predictor at the forecast level and competitive at the forecaster level, though weaker than prior accuracy. Human ratings of rationale quality are less consistently correlated with accuracy and place disproportionate weight on rationale length. Results transfer to an independent forecasting study. EQMs provide a scalable, interpretable method for extracting judgment-relevant information from written explanations. ## Recorded benchmark results A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage. - [Accuracy Correlation Analysis on ACE Forecaster-level Season 4 1.0](): Accuracy Correlation Analysis; ACE Forecaster-level Season 4 1.0; Human Rating Correlation Coefficient: 0.4. - [Accuracy Correlation Analysis on ACE Forecast-level Season 4 1.0](): Accuracy Correlation Analysis; ACE Forecast-level Season 4 1.0; Human Rating r: 0.07. - [Forecast Accuracy Correlation Analysis on ACE Forecast Level (main analysis sample)](): Forecast accuracy correlation analysis; ACE Forecast Level (main analysis sample); Pre-LLM Correlation (r): 0.06. - [Forecaster Accuracy Correlation Analysis on ACE Forecaster Level (main analysis sample)](): Forecaster accuracy correlation analysis; ACE Forecaster Level (main analysis sample); Pre-LLM Correlation (r): 0.39. - [Forecasting Accuracy Correlation on Team Dynamics (Forecast level)](): Forecasting Accuracy Correlation; Team Dynamics (Forecast level); Correlation (r) Between: 0.26. - [Forecasting Accuracy Correlation on Team Dynamics (Forecaster level)](): Forecasting Accuracy Correlation; Team Dynamics (Forecaster level); r (Between): 0.46. - [Forecast-level Accuracy Correlation on Forecasting Indicators Dataset Season 3 to 4 (S3→S4)](): Forecast-level Accuracy Correlation; Forecasting Indicators Dataset Season 3 to 4 (S3→S4); Pearson Correlation (r): 0.57. - [Forecast-level Accuracy Correlation on Forecasting Indicators Dataset Season 1 to 2 (S1→S2)](): Forecast-level Accuracy Correlation; Forecasting Indicators Dataset Season 1 to 2 (S1→S2); Pearson Correlation (r): 0.55. - [Forecast-level Accuracy Correlation on Forecasting Indicators Dataset Mean across seasons Aggregate](): Forecast-level Accuracy Correlation; Forecasting Indicators Dataset Mean across seasons Aggregate; Pearson Correlation (r): 0.55. - [Forecast-level Accuracy Correlation on Forecasting Indicators Dataset S2→S3 Season 2 to 3](): Forecast-level Accuracy Correlation; Forecasting Indicators Dataset S2→S3 Season 2 to 3; Pearson Correlation (r): 0.53. ## Original sources and implementation [Original paper]() [PDF]()