# DECIDER: Leveraging Foundation Model Priors for Improved Model Failure Detection and Explanation

Source: [DECIDER: Leveraging Foundation Model Priors for Improved Model Failure Detection and Explanation](<https://www.sota2.com/research/paper/68fa744f1e209131cea60f12>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2025-10-27T06:37:02.192Z

Authors: Rakshith Subramanyam, Kowshik Thopalli, Vivek Narayanaswamy, Jayaraman J.Thiagarajan

Published: 2024-08-01T07:08:11.000Z

## Abstract

Reliably detecting when a deployed machine learning model is likely to fail on a given input is crucial for ensuring safe operation. In this work, we propose DECIDER (Debiasing Classifiers to Identify Errors Reliably), a novel approach that leverages priors from large language models (LLMs) and vision-language models (VLMs) to detect failures in image classification models. DECIDER utilizes LLMs to specify task-relevant core attributes and constructs a \`\`debiased'' version of the classifier by aligning its visual features to these core attributes using a VLM, and detects potential failure by measuring disagreement between the original and debiased models. In addition to proactively identifying samples on which the model would fail, DECIDER also provides human-interpretable explanations for failure through a novel attribute-ablation strategy. Through extensive experiments across diverse benchmarks spanning subpopulation shifts (spurious correlations, class imbalance) and covariate shifts (synthetic corruptions, domain shifts), DECIDER consistently achieves state-of-the-art failure detection performance, significantly outperforming baselines in terms of the overall Matthews correlation coefficient as well as failure and success recall. Our codes can be accessed at~\\url{https://github.com/kowshikthopalli/DECIDER/}

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2408.00331>)

[PDF](<https://arxiv.org/pdf/2408.00331.pdf>)
