# Increasing Trustworthiness of Deep Neural Networks via Accuracy Monitoring Source: [Increasing Trustworthiness of Deep Neural Networks via Accuracy Monitoring]() Collection index: [llms.txt]() Data updated: 2026-01-04T03:16:52.279Z Authors: Zhihui Shao, Jianyi Yang, Shaolei Ren Published: 2020-07-03T03:09:36.000Z ## Abstract Inference accuracy of deep neural networks (DNNs) is a crucial performance metric, but can vary greatly in practice subject to actual test datasets and is typically unknown due to the lack of ground truth labels. This has raised significant concerns with trustworthiness of DNNs, especially in safety-critical applications. In this paper, we address trustworthiness of DNNs by using post-hoc processing to monitor the true inference accuracy on a user's dataset. Concretely, we propose a neural network-based accuracy monitor model, which only takes the deployed DNN's softmax probability output as its input and directly predicts if the DNN's prediction result is correct or not, thus leading to an estimate of the true inference accuracy. The accuracy monitor model can be pre-trained on a dataset relevant to the target application of interest, and only needs to actively label a small portion (1% in our experiments) of the user's dataset for model transfer. For estimation robustness, we further employ an ensemble of monitor models based on the Monte-Carlo dropout method. We evaluate our approach on different deployed DNN models for image classification and traffic sign detection over multiple datasets (including adversarial samples). The result shows that our accuracy monitor model provides a close-to-true accuracy estimation and outperforms the existing baseline methods. ## Recorded benchmark results A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage. - [Image Classification on ImageNet A 2012 (val)](): Image Classification; ImageNet A 2012 (val); AUPR: 0.7192. - [Accuracy Estimation on GTSD D1 (test)](): Accuracy Estimation; GTSD D1 (test); AUPR: 65.85. - [Accuracy Estimation on GTSD-OOD CIFAR-10 + GTSD-D2](): Accuracy Estimation; GTSD-OOD CIFAR-10 + GTSD-D2; AUPR: 0.909. - [Accuracy Estimation on CINIC-10 (test)](): Accuracy Estimation; CINIC-10 (test); AUPR: 86.45. - [Accuracy Estimation on STL-10 (test)](): Accuracy Estimation; STL-10 (test); AUPR: 79.66. - [Accuracy Estimation on AD-10 (test)](): Accuracy Estimation; AD-10 (test); AUPR: 0.8935. - [Accuracy Estimation on GTSD-D2 augmented dataset](): Accuracy Estimation; GTSD-D2 augmented dataset; AUPR: 84.14. - [Accuracy Estimation on CIFAR-10 (test)](): Accuracy Estimation; CIFAR-10 (test); AUPR: 92.7. - [Image Classification on ImageNet B 2012 (val)](): Image Classification; ImageNet B 2012 (val); Estimated Accuracy: not recorded. ## Original sources and implementation [Original paper]() [PDF]()