Image Classification Calibration on ImageNet
1.39Expected Calibration ErrorVector
Evaluation results
| Method | Links | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| VectorK (Number of clusters)=-, λ (Shrinkage parameter)=-2025.10 | 1.39 | 79.19 | — | 0.8784 | 0.2987 | — | — | — | |
| clustered Vector scalingK (Number of clusters)=10, λ (Shrinkage parameter)=52025.10 | 1.39 | 79.19 | — | 0.8783 | 0.2987 | — | — | — | |
| BalCALBackbone=ViT-B/162025.04 | 1.48 | 82.32 | 1.49 | — | — | — | — | — | |
| DualFocalBackbone=ViT-B/162025.04 | 1.69 | 81.94 | 1.61 | — | — | — | — | — | |
| VTSTBackbone=ViT-B/162025.04 | 2.01 | 80 | 1.91 | — | — | — | — | — | |
| Baseα (alpha) configuration=0.01, Nonconformity score type=APS, Transformation type=adaptive2025.05 | 2.41 | — | — | 0.7 | 0.2606 | 8.92 | 0.43 | 0.4 | |
| Temp. scalingα (alpha) configuration=0.01, Nonconformity score type=APS, Transformation type=adaptive2025.05 | 2.49 | — | — | 0.7 | 0.2606 | 9.08 | 0.42 | 0.39 | |
| Conformal Temperature Scalinglabel=TS (ours), α (alpha) configuration=0.01, Nonconformity score type=APS, Transformation type=adaptive2025.05 | 2.87 | — | — | 0.73 | 0.2635 | 13.34 | 0.34 | 0.01 | |
| Binwise-VectorK (Number of clusters)=-, λ (Shrinkage parameter)=-2025.10 | 3.13 | 75.99 | — | 1.0374 | 0.3534 | — | — | — | |
| clustered Temperature ScalingK (Number of clusters)=50, λ (Shrinkage parameter)=52025.10 | 3.21 | 78.99 | — | 0.872 | 0.3015 | — | — | — | |
| TSK (Number of clusters)=-, λ (Shrinkage parameter)=-2025.10 | 3.27 | 78.99 | — | 0.8721 | 0.3015 | — | — | — | |
| Binwise-TSK (Number of clusters)=-, λ (Shrinkage parameter)=-2025.10 | 3.73 | 78.99 | — | 0.869 | 0.3017 | — | — | — | |
| TSTBackbone=ViT-B/162025.04 | 4.38 | 81.04 | 4.31 | — | — | — | — | — | |
| Isotonicα (alpha) configuration=0.01, Nonconformity score type=APS, Transformation type=adaptive2025.05 | 4.43 | — | — | 2.12 | 0.2736 | 11.35 | 2.56 | 5.71 | |
| MMCEBackbone=ViT-B/162025.04 | 4.91 | 81.68 | 4.86 | — | — | — | — | — | |
| VanillaBackbone=ViT-B/162025.04 | 5.74 | 81.85 | 5.72 | — | — | — | — | — | |
| Naive CMCEα (alpha) configuration=0.01, Nonconformity score type=APS, Transformation type=adaptive2025.05 | 5.82 | — | — | 0.73 | 0.2672 | 16.18 | 0.02 | 0.93 | |
| Ada-TSα (alpha) configuration=0.01, Nonconformity score type=APS, Transformation type=adaptive2025.05 | 10.81 | — | — | 1.69 | 0.3045 | 25.53 | 0.58 | 7.21 | |
| Platt Scalingα (alpha) configuration=0.01, Nonconformity score type=APS, Transformation type=adaptive2025.05 | 12.89 | — | — | 0.81 | 0.2862 | 20.67 | 1.26 | 0.92 | |
| V.-Abers (OvA)α (alpha) configuration=0.01, Nonconformity score type=APS, Transformation type=adaptive2025.05 | 23.93 | — | — | 1 | 0.3332 | 33.66 | 7.92 | 1 | |
| BaseK (Number of clusters)=-, λ (Shrinkage parameter)=-2025.10 | 40.08 | 78.99 | — | 1.4776 | 0.4872 | — | — | — | |
| DirichletK (Number of clusters)=10, λ (Shrinkage parameter)=52025.10 | 71.29 | 0.27 | — | 35.9447 | 1.6116 | — | — | — | |
| DirichletK (Number of clusters)=-, λ (Shrinkage parameter)=-2025.10 | 75.41 | 0.28 | — | 15.8978 | 1.6724 | — | — | — |