Big-Bench Hard Reasoning on BBH n=2395 (test)
100ActivationCRC
Evaluation results
| Method | Links | |||||
|---|---|---|---|---|---|---|
| CRCOperating Budget (T)=4, Scaling parameter (λ*)=4.1, Target budget (β)=.33, Training error (eT)=.082026.06 | 100 | 91.6 | 8.4 | 0.26 | 1.1 | |
| Selective Pred.Operating Budget (T)=4, Scaling parameter (λ*)=4.1, Target budget (β)=.33, Training error (eT)=.082026.06 | 99.9 | 91.7 | 8.3 | 0.25 | 1.1 | |
| Calibrated LearnedOperating Budget (T)=4, Scaling parameter (λ*)=4.1, Target budget (β)=.33, Training error (eT)=.082026.06 | 99.7 | 91.4 | 8.6 | 0.26 | 1 | |
| Isotonic Conf.Operating Budget (T)=4, Scaling parameter (λ*)=4.1, Target budget (β)=.33, Training error (eT)=.082026.06 | 99.7 | 91.7 | 8.3 | 0.25 | 1.1 | |
| Learned StopperOperating Budget (T)=4, Scaling parameter (λ*)=4.1, Target budget (β)=.33, Training error (eT)=.082026.06 | 99.4 | 91.7 | 8.3 | 0.25 | 1.1 | |
| ConsensusOperating Budget (T)=4, Scaling parameter (λ*)=4.1, Target budget (β)=.33, Training error (eT)=.082026.06 | 98.2 | 91.5 | 8.5 | 0.26 | 1.3 | |
| Adaptive Local k-NN Certification (kNN no bias)Operating Budget (T)=4, Scaling parameter (λ*)=4.1, Target budget (β)=.33, Training error (eT)=.082026.06 | 95.7 | 92.7 | 7.1 | 0.22 | 1.2 | |
| OracleOperating Budget (T)=4, Scaling parameter (λ*)=4.1, Target budget (β)=.33, Training error (eT)=.082026.06 | 92.6 | 100 | 0 | 0 | 1.2 | |
| Confidence Thr.Operating Budget (T)=4, Scaling parameter (λ*)=4.1, Target budget (β)=.33, Training error (eT)=.082026.06 | 87.1 | 91.5 | 8.5 | 0.26 | 1.6 | |
| Adaptive Local k-NN CertificationOperating Budget (T)=4, Scaling parameter (λ*)=4.1, Target budget (β)=.33, Training error (eT)=.082026.06 | 84.4 | 96.4 | 3 | 0.09 | 1.8 | |
| Adaptive Local k-NN Certification (Final-round)Operating Budget (T)=4, Scaling parameter (λ*)=4.1, Target budget (β)=.33, Training error (eT)=.082026.06 | 84.4 | 96.4 | 3.1 | 0.1 | 4 |