Selective Classification for Deep Neural Networks

About

Selective classification techniques (also known as reject option) have not yet been considered in the context of deep neural networks (DNNs). These techniques can potentially significantly improve DNNs prediction performance by trading-off coverage. In this paper we propose a method to construct a selective classifier given a trained neural network. Our method allows a user to set a desired risk level. At test time, the classifier rejects instances as needed, to grant the desired risk (with high probability). Empirical results over CIFAR and ImageNet convincingly demonstrate the viability of our method, which opens up possibilities to operate DNNs in mission-critical applications. For example, using our method an unprecedented 2% error in top-5 ImageNet classification can be guaranteed with probability 99.9%, and almost 60% test coverage.

Yonatan Geifman, Ran El-Yaniv · 2017

Related benchmarks

Task	Dataset	Result
Image Classification	CIFAR-100	Accuracy100	375
Unsupervised Domain Adaptation	Office-Home	Average Accuracy22.5	295
Unsupervised Domain Adaptation	ImageCLEF-DA	Average Accuracy7.66	114
Unsupervised Domain Adaptation	VisDA unsupervised domain adaptation 2017	Mean Accuracy32.03	103
Selective Classification	CIFAR-10 (test)	Accuracy100	98
Uncertainty Quantification	SVHN (test)	Accuracy100	80
Graph Classification	PROTEINS	Classification Error Rate3.4	72
Robotic task execution	LIBERO	Average Success Rate69	44
Misclassification Detection	COLA	ROC-AUC76.9	31
Active Learning Regression	PartMC target χa (test)	R^20.758	22