Fragility-aware Classification for Understanding Risk and Improving Generalization
About
Classification models play a central role in data-driven decision-making applications such as medical diagnosis, recommendation systems, and risk assessment. Traditional performance metrics, such as accuracy and AUC, focus on overall error rates but fail to account for the confidence of incorrect predictions, i.e., the risk of confident misjudgments. This limitation is particularly consequential in safety-critical and cost-sensitive settings, where overconfident errors can lead to severe outcomes. To address this issue, we propose the Fragility Index (FI), a novel performance metric that evaluates classifiers from a risk-averse perspective by capturing the tail risk of confident misjudgments. We formulate FI within a robust satisficing (RS) framework to ensure robustness under distributional uncertainty. Building on this, we develop a tractable training framework that directly targets FI via a surrogate loss, and show that models trained under this framework admit provable bounds on FI. We further derive exact reformulations for a broad class of loss functions, including cross-entropy, hinge-type, and Lipschitz losses, and extend the approach to deep neural networks. Empirical results on real-world medical diagnosis tasks demonstrate that FI complements existing metrics by revealing error tail risk and improving decision quality. FI-based models achieve competitive accuracy and AUC while consistently reducing confident misjudgments and associated operational costs, offering a practical tool for improving robustness and reliability in risk-critical applications.