Out-of-distribution Detection on CIFAR-10 (ID) vs TinyImageNet (OOD) (test)
99.88AUROCDeep Abstaining Classifier
Evaluation results
| Method | Links | |||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Deep Abstaining ClassifierBackbone=ResNet 342021.05 | 99.88 | — | — | — | — | 0.36 | — | — | — | — | — | — | — | |
| OursBackbone=ResNet2019.12 | 99.72 | 99.48 | 97.82 | — | — | — | — | — | — | — | — | — | — | |
| OursBackbone=ResNet2019.12 | 99.7 | 98.7 | 97.8 | — | — | — | — | — | — | — | — | — | — | |
| OursBackbone=DenseNet2019.12 | 99.7 | 98.8 | 97.9 | — | — | — | — | — | — | — | — | — | — | |
| OursModel Architecture=ResNet2019.12 | 99.7 | 98.7 | 97.8 | — | — | — | — | — | — | — | — | — | — | |
| OursModel Architecture=DenseNet2019.12 | 99.7 | 98.8 | 97.9 | — | — | — | — | — | — | — | — | — | — | |
| Mahalanobis distance-based scoreBackbone=ResNet, Validation Setup=OOD samples2018.07 | 99.5 | 97.1 | 96.3 | — | — | — | — | — | — | — | — | — | — | |
| MahalanobisBackbone=ResNet2018.07 | 99.5 | 97.1 | 96.3 | 99.5 | 99.5 | — | — | — | — | — | — | — | — | |
| MahalanobisBackbone=ResNet2019.12 | 99.5 | 97.1 | 96.3 | — | — | — | — | — | — | — | — | — | — | |
| MahalanobisBackbone=ResNet2019.12 | 99.5 | 97.1 | 96.3 | — | — | — | — | — | — | — | — | — | — | |
| MahalanobisModel Architecture=ResNet2019.12 | 99.5 | 97.1 | 96.3 | — | — | — | — | — | — | — | — | — | — | |
| Deep Abstaining ClassifierBackbone=Wide ResNet 28x102021.05 | 99.45 | — | — | — | — | 1.91 | — | — | — | — | — | — | — | |
| Ensemble of Leave-out ClassifiersBackbone=Wide ResNet 28x102021.05 | 99.36 | — | — | — | — | 2.94 | — | — | — | — | — | — | — | |
| OursModel Architecture=DenseNet2019.12 | 99.3 | 96.7 | 96.1 | — | — | — | — | — | — | — | — | — | — | |
| OursModel Architecture=ResNet2019.12 | 99.2 | 96.7 | 96.1 | — | — | — | — | — | — | — | — | — | — | |
| ODINArchitecture=Dense-BC, Temperature (T)=1000, Noise magnitude (epsilon)=0.00142017.06 | 99.1 | — | — | 99.1 | 99.1 | 4.3 | 4.7 | — | — | — | — | — | — | |
| ODIN_OODbaseline=ODIN2021.06 | 99.1 | — | — | — | — | 4.3 | — | — | — | — | — | — | — | |
| Mahalanobis distance-based scoreBackbone=ResNet, Validation Setup=adversarial samples2018.07 | 99 | 95.5 | 95.4 | — | — | — | — | — | — | — | — | — | — | |
| Deep Mahalanobis DetectorBackbone=ResNet 342021.05 | 99 | — | — | — | — | 4.5 | — | — | — | — | — | — | — | |
| D_alphadetector=DOCTOR, variant=alpha2021.06 | 98.9 | — | — | — | — | 4.6 | — | — | — | — | — | — | — | |
| Mahalanobis distance-based scoreBackbone=DenseNet, Validation Setup=OOD samples2018.07 | 98.8 | 95 | 95 | — | — | — | — | — | — | — | — | — | — | |
| Mahalanobis distance-based scoreBackbone=DenseNet, Validation Setup=adversarial samples2018.07 | 98.8 | 94.9 | 95 | — | — | — | — | — | — | — | — | — | — | |
| MahalanobisBackbone=DenseNet2018.07 | 98.8 | 95 | 95 | 98.8 | 98.8 | — | — | — | — | — | — | — | — | |
| MahalanobisBackbone=DenseNet2019.12 | 98.8 | 95 | 95 | — | — | — | — | — | — | — | — | — | — | |
| MahalanobisModel Architecture=DenseNet2019.12 | 98.8 | 95 | 95 | — | — | — | — | — | — | — | — | — | — | |
| MahalanobisModel Architecture=ResNet2019.12 | 98.6 | 92 | 93.9 | — | — | — | — | — | — | — | — | — | — | |
| ODINBackbone=DenseNet, Validation Setup=OOD samples2018.07 | 98.5 | 92.4 | 93.9 | — | — | — | — | — | — | — | — | — | — | |
| ODINBackbone=DenseNet2018.07 | 98.5 | 92.4 | 93.9 | 98.5 | 98.5 | — | — | — | — | — | — | — | — | |
| ODINBackbone=DenseNet2019.12 | 98.5 | 92.4 | 93.9 | — | — | — | — | — | — | — | — | — | — | |
| ODINModel Architecture=DenseNet2019.12 | 98.5 | 92.4 | 93.9 | — | — | — | — | — | — | — | — | — | — | |
| ODIN_OODbaseline=ODIN2021.06 | 98.5 | — | — | — | — | 7.2 | — | — | — | — | — | — | — | |
| D_betadetector=DOCTOR, variant=beta2021.06 | 98.5 | — | — | — | — | 6.4 | — | — | — | — | — | — | — | |
| ODINModel Architecture=DenseNet2019.12 | 97.6 | 87 | 92.3 | — | — | — | — | — | — | — | — | — | — | |
| D_alphadetector=DOCTOR, variant=alpha2021.06 | 97.6 | — | — | — | — | 9.9 | — | — | — | — | — | — | — | |
| D_betadetector=DOCTOR, variant=beta2021.06 | 97.3 | — | — | — | — | 11.2 | — | — | — | — | — | — | — | |
| ODINBackbone=DenseNet, Validation Setup=adversarial samples2018.07 | 97.2 | 87.1 | 92.1 | — | — | — | — | — | — | — | — | — | — | |
| VDBackbone=ResNet2019.12 | 96.8 | 82.9 | 91.3 | — | — | — | — | — | — | — | — | — | — | |
| AHGCBackbone=ResNet-18, Optimizer=SGD2024.12 | 96.45 | — | — | 97.94 | 92.96 | 9.71 | — | — | — | 72.84 | 81.85 | 85.22 | 86.67 | |
| MahalanobisModel Architecture=DenseNet2019.12 | 95.3 | 84.2 | 89.9 | — | — | — | — | — | — | — | — | — | — | |
| Baseline (Hendrycks & Gimpel, 2017)Architecture=Dense-BC, Temperature (T)=1, Noise magnitude (epsilon)=02017.06 | 95.3 | — | — | 96.4 | 93.8 | 34.7 | 10 | — | — | — | — | — | — | |
| NGCIND noise=50% symmetric, OOD training samples=20k, OOD test samples=10k2021.08 | 94.18 | — | 93.54 | — | — | — | — | — | 87.5 | — | — | — | — | |
| BaselineBackbone=DenseNet, Validation Setup=OOD samples2018.07 | 94.1 | 58.9 | 88.5 | — | — | — | — | — | — | — | — | — | — | |
| BaselineBackbone=DenseNet, Validation Setup=adversarial samples2018.07 | 94.1 | 58.9 | 88.5 | — | — | — | — | — | — | — | — | — | — | |
| BaselineBackbone=DenseNet2018.07 | 94.1 | 58.9 | 88.5 | 95.3 | 92.3 | — | — | — | — | — | — | — | — | |
| BaselineBackbone=DenseNet2019.12 | 94.1 | 58.9 | 88.5 | — | — | — | — | — | — | — | — | — | — | |
| BaselineModel Architecture=DenseNet2019.12 | 94.1 | 58.9 | 88.5 | — | — | — | — | — | — | — | — | — | — | |
| ODINBackbone=ResNet, Validation Setup=OOD samples2018.07 | 94 | 72.5 | 86.5 | — | — | — | — | — | — | — | — | — | — | |
| ODINBackbone=ResNet2018.07 | 94 | 72.5 | 86.5 | 94.2 | 94.1 | — | — | — | — | — | — | — | — | |
| ODINBackbone=ResNet2019.12 | 94 | 67.9 | 86.5 | — | — | — | — | — | — | — | — | — | — | |
| ODINBackbone=ResNet2019.12 | 94 | 67.9 | 86.5 | — | — | — | — | — | — | — | — | — | — | |
| ODINModel Architecture=ResNet2019.12 | 94 | 67.9 | 86.5 | — | — | — | — | — | — | — | — | — | — | |
| ODINBackbone=ResNet, Validation Setup=adversarial samples2018.07 | 93.9 | 69.6 | 86 | — | — | — | — | — | — | — | — | — | — | |
| BaselineModel Architecture=DenseNet2019.12 | 93.8 | 56.7 | 88.1 | — | — | — | — | — | — | — | — | — | — | |
| SconeBackbone=ResNet-18, Optimizer=SGD2024.12 | 93.48 | — | — | 95.34 | 89.59 | 55.62 | — | — | — | 10.76 | 21.4 | 62.55 | 81.64 | |
| ProtoMixIND noise=50% symmetric, OOD training samples=20k, OOD test samples=10k2021.08 | 93.47 | — | 93.12 | — | — | — | — | — | 86.2 | — | — | — | — | |
| OpenMaxBackbone=ResNet 342021.05 | 93.39 | — | — | — | — | 24.2 | — | — | — | — | — | — | — | |
| ODINOODProtocol=PBB, Model Setup=♠2021.06 | 93.2 | — | — | — | — | — | — | — | — | — | — | — | — | |
| ODINModel Architecture=ResNet2019.12 | 93.1 | 68.7 | 85.2 | — | — | — | — | — | — | — | — | — | — | |
| DPNBackbone=ResNet2019.12 | 93 | 71.6 | 86.4 | — | — | — | — | — | — | — | — | — | — | |
| ODINBackbone=Wide ResNet 28x102021.05 | 92.1 | — | — | — | — | 25.5 | — | — | — | — | — | — | — | |
| UDGBackbone=ResNet-18, Optimizer=SGD2024.12 | 91.91 | — | — | 94.43 | 86.99 | 50.18 | — | — | — | 0.32 | 23.15 | 53.96 | 78.36 | |
| BaselineModel Architecture=ResNet2019.12 | 91.4 | 46.4 | 85.4 | — | — | — | — | — | — | — | — | — | — | |
| BaselineBackbone=ResNet, Validation Setup=OOD samples2018.07 | 91 | 44.7 | 85.1 | — | — | — | — | — | — | — | — | — | — | |
| BaselineBackbone=ResNet, Validation Setup=adversarial samples2018.07 | 91 | 44.7 | 85.1 | — | — | — | — | — | — | — | — | — | — | |
| BaselineBackbone=ResNet2018.07 | 91 | 44.7 | 85.1 | 92.5 | 88.4 | — | — | — | — | — | — | — | — | |
| BaselineBackbone=ResNet2019.12 | 91 | 41 | 85.1 | — | — | — | — | — | — | — | — | — | — | |
| BaselineBackbone=ResNet2019.12 | 91 | 41 | 85.1 | — | — | — | — | — | — | — | — | — | — | |
| BaselineModel Architecture=ResNet2019.12 | 91 | 41 | 85.1 | — | — | — | — | — | — | — | — | — | — | |
| ConjNormBackbone=ResNet-18, Optimizer=SGD2024.12 | 90.74 | — | — | 83.25 | 80.24 | 52.8 | — | — | — | 9.52 | 19.37 | 59.88 | 79.45 | |
| OEBackbone=ResNet-18, Optimizer=SGD2024.12 | 87.65 | — | — | 90.9 | 82.16 | 58.98 | — | — | — | 14.37 | 18.84 | 33.65 | 66.03 | |
| EBOBackbone=ResNet-18, Optimizer=SGD2024.12 | 81.65 | — | — | 81.8 | 78.75 | 57.81 | — | — | — | 0.33 | 0.95 | 6.01 | 40.4 | |
| MCDBackbone=ResNet-18, Optimizer=SGD2024.12 | 80.98 | — | — | 85.63 | 72.48 | 78.89 | — | — | — | 1.62 | 4.15 | 19.37 | 56.08 | |
| ODINBackbone=ResNet-18, Optimizer=SGD2024.12 | 79.69 | — | — | 79.34 | 77.52 | 59.09 | — | — | — | 0.36 | 0.63 | 4.49 | 34.52 | |
| BNN-ARHTArchitecture=LeNet2023.10 | 67.77 | — | — | — | — | — | — | 66.74 | — | — | — | — | — | |
| MC DropoutArchitecture=LeNet2023.10 | 66.98 | — | — | — | — | — | — | 64.46 | — | — | — | — | — | |
| Deep EnsemblesArchitecture=LeNet2023.10 | 66.41 | — | — | — | — | — | — | 63.97 | — | — | — | — | — | |
| Kendall and GalArchitecture=LeNet2023.10 | 63.23 | — | — | — | — | — | — | 63.06 | — | — | — | — | — | |
| DPNArchitecture=LeNet2023.10 | 61.68 | — | — | — | — | — | — | 58.33 | — | — | — | — | — | |
| EDLArchitecture=LeNet2023.10 | 51.64 | — | — | — | — | — | — | 66.31 | — | — | — | — | — |