# PIE: Performance Interval Estimation for Free-Form Generation Tasks

Source: [PIE: Performance Interval Estimation for Free-Form Generation Tasks](<https://www.sota2.com/research/paper/68fa7b2b1e209131cea86f82>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2026-01-15T05:00:12.598Z

Authors: Chi-Yang Hsu, Alexander Braylan, Yiheng Su, Matthew Lease, Omar Alonso

Published: 2025-09-09T00:59:34.000Z

## Abstract

Confidence estimation infers a probability for whether each model output is correct or not. While predicting such binary correctness is sensible for tasks with exact answers, free-form generation tasks are often more nuanced, with output quality being both fine-grained and multi-faceted. We thus propose Performance Interval Estimation (PIE) to predict both: 1) point estimates for any arbitrary set of continuous-valued evaluation metrics; and 2) calibrated uncertainty intervals around these point estimates. We then compare two approaches: LLM-as-judge vs. classic regression with confidence estimation features. Evaluation over 11 datasets spans summarization, translation, code generation, function-calling, and question answering. Regression is seen to achieve both: i) lower error point estimates of metric scores; and ii) well-calibrated uncertainty intervals. To support reproduction and follow-on work, we share our data and code.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2509.07309>)

[PDF](<https://arxiv.org/pdf/2509.07309.pdf>)
