Multistep Soft Reasoning on MuSR n=756 (test)
100Activation (Act)Adaptive Local k-NN Certification (kNN no bias)
Evaluation results
| Method | Links | |||||
|---|---|---|---|---|---|---|
| Adaptive Local k-NN Certification (kNN no bias)Operating Budget (T)=4, Scaling parameter (λ*)=2.2, Target budget (β)=.76, Training error (eT)=.342026.06 | 100 | 64.4 | 35.6 | 47 | 1 | |
| CRCOperating Budget (T)=4, Scaling parameter (λ*)=2.2, Target budget (β)=.76, Training error (eT)=.342026.06 | 100 | 64.3 | 35.7 | 47 | 1 | |
| Selective Pred.Operating Budget (T)=4, Scaling parameter (λ*)=2.2, Target budget (β)=.76, Training error (eT)=.342026.06 | 100 | 64.6 | 35.5 | 47 | 1 | |
| Calibrated LearnedOperating Budget (T)=4, Scaling parameter (λ*)=2.2, Target budget (β)=.76, Training error (eT)=.342026.06 | 100 | 64.4 | 35.6 | 47 | 1 | |
| Isotonic Conf.Operating Budget (T)=4, Scaling parameter (λ*)=2.2, Target budget (β)=.76, Target error (eT)=.342026.06 | 100 | 64.7 | 35.3 | 46 | 1 | |
| Learned StopperOperating Budget (T)=4, Scaling parameter (λ*)=2.2, Target budget (β)=.76, Training error (eT)=.342026.06 | 98.8 | 63.5 | 36.5 | 48 | 1.1 | |
| ConsensusOperating Budget (T)=4, Scaling parameter (λ*)=2.2, Target budget (β)=.76, Training error (eT)=.342026.06 | 93.1 | 66.8 | 33.2 | 44 | 1.6 | |
| OracleOperating Budget (T)=4, Scaling parameter (λ*)=2.2, Target budget (β)=.76, Training error (eT)=.342026.06 | 69.8 | 100 | 0 | 0 | 2 | |
| Adaptive Local k-NN Certification (Final-round)Operating Budget (T)=4, Scaling parameter (λ*)=2.2, Target budget (β)=.76, Training error (eT)=.342026.06 | 46.3 | 69.3 | 14.3 | 19 | 4 | |
| Confidence Thr.Operating Budget (T)=4, Scaling parameter (λ*)=2.2, Target budget (β)=.76, Training error (eT)=.342026.06 | 32.3 | 67.2 | 32.8 | 43 | 3.3 | |
| Adaptive Local k-NN CertificationOperating Budget (T)=4, Scaling parameter (λ*)=2.2, Target budget (β)=.76, Training error (eT)=.342026.06 | 9 | 70.3 | 2.7 | 4 | 3.8 |