# Measuring Forecasting Skill from Text Source: [Measuring Forecasting Skill from Text]() Collection index: [llms.txt]() Data updated: 2025-10-25T22:20:05.971Z Authors: Shi Zong, Alan Ritter, Eduard Hovy Published: 2020-06-12T19:04:10.000Z ## Abstract People vary in their ability to make accurate predictions about the future. Prior studies have shown that some individuals can predict the outcome of future events with consistently better accuracy. This leads to a natural question: what makes some forecasters better than others? In this paper we explore connections between the language people use to describe their predictions and their forecasting skill. Datasets from two different forecasting domains are explored: (1) geopolitical forecasts from Good Judgment Open, an online prediction forum and (2) a corpus of company earnings forecasts made by financial analysts. We present a number of linguistic metrics which are computed over text associated with people's predictions about the future including: uncertainty, readability, and emotion. By studying linguistic factors associated with predictions, we are able to shed some light on the approach taken by skilled forecasters. Furthermore, we demonstrate that it is possible to accurately predict forecasting skill using a model that is based solely on language. This could potentially be useful for identifying accurate predictions or potentially skilled forecasters earlier. ## Recorded benchmark results A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage. - [Forecast Accuracy Correlation Analysis on ACE Forecast Level (main analysis sample)](): Forecast accuracy correlation analysis; ACE Forecast Level (main analysis sample); Pre-LLM Correlation (r): 0.06. - [Forecaster Accuracy Correlation Analysis on ACE Forecaster Level (main analysis sample)](): Forecaster accuracy correlation analysis; ACE Forecaster Level (main analysis sample); Pre-LLM Correlation (r): 0.39. - [Forecast-level Accuracy Correlation on Forecasting Indicators Dataset Season 3 to 4 (S3→S4)](): Forecast-level Accuracy Correlation; Forecasting Indicators Dataset Season 3 to 4 (S3→S4); Pearson Correlation (r): 0.52. - [Forecast-level Accuracy Correlation on Forecasting Indicators Dataset Season 1 to 2 (S1→S2)](): Forecast-level Accuracy Correlation; Forecasting Indicators Dataset Season 1 to 2 (S1→S2); Pearson Correlation (r): 0.35. - [Forecast-level Accuracy Correlation on Forecasting Indicators Dataset S2→S3 Season 2 to 3](): Forecast-level Accuracy Correlation; Forecasting Indicators Dataset S2→S3 Season 2 to 3; Pearson Correlation (r): 0.41. - [Forecast-level Accuracy Correlation on Forecasting Indicators Dataset Mean across seasons Aggregate](): Forecast-level Accuracy Correlation; Forecasting Indicators Dataset Mean across seasons Aggregate; Pearson Correlation (r): 0.43. ## Original sources and implementation [Original paper]() [PDF]()