RISE: Randomized Input Sampling for Explanation of Black-box Models
About
Deep neural networks are being used increasingly to automate data analysis and decision making, yet their decision-making process is largely unclear and is difficult to explain to the end users. In this paper, we address the problem of Explainable AI for deep neural networks that take images as input and output a class probability. We propose an approach called RISE that generates an importance map indicating how salient each pixel is for the model's prediction. In contrast to white-box approaches that estimate pixel importance using gradients or other internal network state, RISE works on black-box models. It estimates importance empirically by probing the model with randomly masked versions of the input image and obtaining the corresponding outputs. We compare our approach to state-of-the-art importance extraction methods using both an automatic deletion/insertion metric and a pointing metric based on human-annotated object segments. Extensive experiments on several benchmark datasets show that our approach matches or exceeds the performance of other methods, including white-box approaches. Project page: http://cs-people.bu.edu/vpetsiuk/rise/
Key insights
What problem does RISE solve?
RISE explains black-box image classifiers by generating pixel importance maps. It probes models with randomly masked images and weights masks by output scores, requiring no access to model internals like gradients or weights.
How does RISE differ from prior work?
Unlike white-box methods needing gradients or features, RISE needs only input-output access. On ImageNet with ResNet50, RISE achieves 0.1076 deletion and 0.7267 insertion AUC, outperforming Grad-CAM and LIME.
What are the practical uses of RISE?
RISE helps debug model errors, revealing that ResNet50 confuses black sheep with cows. It extends to image captioning models, generating per-word importance maps without architectural access, demonstrated on MSCOCO.
What limitations do the authors acknowledge?
RISE is computationally heavy due to many forward passes and can produce noisy importance maps from sampling approximation, especially with varying object sizes. Reducing mask count is left as future work.