Multi-task Language Understanding on MMLU-Pro n=8312 (test)
100ActivationCRC
Evaluation results
| Method | Links | |||||
|---|---|---|---|---|---|---|
| CRCOperating Budget (T)=4, Scaling parameter (λ*)=2.4, Target budget (β)=.38, Training error (eT)=.162026.06 | 100 | 79.9 | 20.1 | 53 | 1 | |
| Isotonic Conf.Operating Budget (T)=4, Scaling parameter (λ*)=2.4, Target budget (β)=.38, Training error (eT)=.162026.06 | 98.3 | 84.4 | 15.4 | 40 | 1.3 | |
| Selective Pred.Operating Budget (T)=4, Scaling parameter (λ*)=2.4, Target budget (β)=.38, Training error (eT)=.162026.06 | 97.4 | 85 | 14.6 | 38 | 1.3 | |
| Calibrated LearnedOperating Budget (T)=4, Scaling parameter (λ*)=2.4, Target budget (β)=.38, Training error (eT)=.162026.06 | 97.3 | 84.5 | 15.1 | 40 | 1.3 | |
| Learned StopperOperating Budget (T)=4, Scaling parameter (λ*)=2.4, Target budget (β)=.38, Training error (eT)=.162026.06 | 96.6 | 83.6 | 16.4 | 43 | 1.3 | |
| ConsensusOperating Budget (T)=4, Scaling parameter (λ*)=2.4, Target budget (β)=.38, Training error (eT)=.162026.06 | 94.5 | 84.5 | 15.5 | 41 | 1.7 | |
| OracleOperating Budget (T)=4, Scaling parameter (λ*)=2.4, Target budget (β)=.38, Training error (eT)=.162026.06 | 86 | 100 | 0 | 0 | 1.5 | |
| Adaptive Local k-NN Certification (kNN no bias)Operating Budget (T)=4, Scaling parameter (λ*)=2.4, Target budget (β)=.38, Training error (eT)=.162026.06 | 84.3 | 90.4 | 8.1 | 21 | 1.6 | |
| Adaptive Local k-NN CertificationOperating Budget (T)=4, Scaling parameter (λ*)=2.4, Target budget (β)=.38, Training error (eT)=.162026.06 | 71.1 | 93.9 | 4.4 | 12 | 2.2 | |
| Confidence Thr.Operating Budget (T)=4, Scaling parameter (λ*)=2.4, Target budget (β)=.38, Training error (eT)=.162026.06 | 69.8 | 84.5 | 15.5 | 41 | 2.4 | |
| Adaptive Local k-NN Certification (Final-round)Operating Budget (T)=4, Scaling parameter (λ*)=2.4, Target budget (β)=.38, Training error (eT)=.162026.06 | 69.1 | 94.4 | 3.9 | 10 | 4 |