Adaptive Cumulative Mass Calibration with Conformal Prediction
About
Reliable probability estimates by classifiers are essential in high-risk applications. In practice, however, predicted probabilities are often miscalibrated, and many existing post-hoc calibration methods typically lack guarantees that a specific notion of calibration is achieved after the correction procedure is applied. We introduce a *set-based* perspective on calibration through the notion of *cumulative mass calibration* and the corresponding error measures. We propose a new calibration procedure based on conformal prediction that forms cumulative probabilities with guaranteed marginal coverage. We introduce an __adaptive temperature scaling algorithm__, with the temperature tuned for each input to satisfy the conformal coverage constraint. As we show, this procedure can be efficiently implemented. Across image classification tasks, particularly in settings with many classes, our method improves newly introduced calibration error measures (__CMCE__ and $\alpha$-CMCE) *and* standard metrics (such as ECE, cw-ECE, MCE) over the existing baselines.