Logical Reasoning on LogiQA n=8678 (test)
100Activation ScoreCRC
Evaluation results
| Method | Links | |||||
|---|---|---|---|---|---|---|
| CRCOperating Budget (T)=4, Scaling parameter (λ*)=1.3, Target budget (β)=.31, Training error (eT)=.242026.06 | 100 | 74.4 | 25.6 | 82 | 1 | |
| Isotonic Conf.Operating Budget (T)=4, Scaling parameter (λ*)=1.3, Target budget (β)=.31, Training error (eT)=.242026.06 | 99.1 | 75.8 | 24 | 77 | 1.1 | |
| Selective Pred.Operating Budget (T)=4, Scaling parameter (λ*)=1.3, Target budget (β)=.31, Training error (eT)=.242026.06 | 98.6 | 76.1 | 23.6 | 76 | 1.1 | |
| Calibrated LearnedOperating Budget (T)=4, Scaling parameter (λ*)=1.3, Target budget (β)=.31, Training error (eT)=.242026.06 | 97.9 | 76.1 | 23.4 | 75 | 1.2 | |
| Learned StopperOperating Budget (T)=4, Scaling parameter (λ*)=1.3, Target budget (β)=.31, Training error (eT)=.242026.06 | 96.6 | 83.6 | 16.4 | 53 | 1.3 | |
| ConsensusOperating Budget (T)=4, Scaling parameter (λ*)=1.3, Target budget (β)=.31, Training error (eT)=.242026.06 | 93.6 | 76 | 24 | 77 | 1.7 | |
| OracleOperating Budget (T)=4, Scaling parameter (λ*)=1.3, Target budget (β)=.31, Training error (eT)=.242026.06 | 78.6 | 100 | 0 | 0 | 1.7 | |
| Adaptive Local k-NN Certification (kNN no bias)Operating Budget (T)=4, Scaling parameter (λ*)=1.3, Target budget (β)=.31, Training error (eT)=.242026.06 | 61.5 | 85.6 | 8.9 | 29 | 2.4 | |
| Confidence Thr.Operating Budget (T)=4, Scaling parameter (λ*)=1.3, Target budget (β)=.31, Training error (eT)=.242026.06 | 33.3 | 76 | 24 | 77 | 2.5 | |
| Adaptive Local k-NN CertificationOperating Budget (T)=4, Scaling parameter (λ*)=1.3, Target budget (β)=.31, Training error (eT)=.242026.06 | 27.9 | 88.9 | 3.2 | 10 | 3.5 | |
| Adaptive Local k-NN Certification (Final-round)Operating Budget (T)=4, Scaling parameter (λ*)=1.3, Target budget (β)=.31, Training error (eT)=.242026.06 | 27.2 | 89 | 3.1 | 10 | 4 |