Selective Prediction on Banking77 ncal=6,468, delta=0.10, simulated confidence scores (test)
90.6Accuracy (alpha=0.15)Clopper-Pearson + LTT
Evaluation results
| Method | Links | ||||||
|---|---|---|---|---|---|---|---|
| Clopper-Pearson + LTTdelta=0.10, n_cal=6,4682026.03 | 90.6 | 25.2 | 43.4 | 64 | 81.8 | 96.8 | |
| WSR Betting + LTTdelta=0.10, n_cal=6,4682026.03 | 90.1 | 28.8 | 47.2 | 65.5 | 81.2 | 97.6 | |
| LTT + Hoeffdingdelta=0.10, n_cal=6,4682026.03 | 89.7 | — | 23.4 | 59.2 | 80.2 | 96.5 | |
| PAC-Bayes-lambdadelta=0.10, n_cal=6,4682026.03 | 88.3 | 21.9 | 39 | 59.2 | 79.2 | 95.7 | |
| Hoeffding + Uniondelta=0.10, n_cal=6,4682026.03 | 87.8 | — | — | 52.2 | 77.3 | 95.7 | |
| Wassersteinepsilon=0.01, delta=0.10, n_cal=6,4682026.03 | 86.6 | — | — | 43.4 | 73.7 | 94.2 | |
| CVaRbeta=0.20, delta=0.10, n_cal=6,4682026.03 | 23.4 | — | — | — | — | 43.4 |