Code Verification on MBPP (test)
100Mean Hit RateOracle
Evaluation results
| Method | Links | ||
|---|---|---|---|
| OracleModel=Qwen3-8B, beta=10%, err=.558, rho=.1782026.06 | 100 | — | |
| Cost-Stratified ThresholdingModel=LLaMA3-8B, beta=10%, err=.644, rho=.3012026.06 | 88.2 | 17 | |
| Cost-Stratified ThresholdingModel=LLaMA3-8B, beta=20%, err=.644, rho=.3012026.06 | 83.7 | 15.1 | |
| Cost-Stratified ThresholdingModel=LLaMA3-8B, beta=30%, err=.644, rho=.3012026.06 | 80.3 | 11 | |
| Cost-Stratified ThresholdingModel=LLaMA3-8B, beta=50%, err=.644, rho=.3012026.06 | 72.8 | 5.3 | |
| Cost-Stratified ThresholdingModel=Qwen3-8B, beta=10%, err=.558, rho=.1782026.06 | 67.1 | 3.4 | |
| Cost-Stratified ThresholdingModel=Qwen3-8B, beta=20%, err=.558, rho=.1782026.06 | 66.1 | 5.1 | |
| Cost-Stratified ThresholdingModel=Qwen3-8B, beta=30%, err=.558, rho=.1782026.06 | 66.1 | 5.6 | |
| MP-StratModel=Qwen3-8B, beta=10%, err=.558, rho=.1782026.06 | 65.6 | — | |
| ThresholdingModel=Qwen3-8B, beta=10%, err=.558, rho=.1782026.06 | 63.6 | — | |
| Cost-Stratified ThresholdingModel=Qwen3-8B, beta=50%, err=.558, rho=.1782026.06 | 63.1 | 4.3 | |
| C+MModel=Qwen3-8B, beta=10%, err=.558, rho=.1782026.06 | 62.2 | — | |
| Cost-Stratified ThresholdingModel=GPT-4o-mini, beta=30%, err=.528, rho=.1742026.06 | 61.4 | 6.7 | |
| Cost-Stratified ThresholdingModel=GPT-4o-mini, beta=50%, err=.528, rho=.1742026.06 | 61.2 | 3.4 | |
| Cost-Stratified ThresholdingModel=GPT-4o-mini, beta=20%, err=.528, rho=.1742026.06 | 61 | 6.5 | |
| MP-AdaptModel=Qwen3-8B, beta=10%, err=.558, rho=.1782026.06 | 59.7 | — | |
| Cost-Stratified ThresholdingModel=GPT-4o-mini, beta=10%, err=.528, rho=.1742026.06 | 59.2 | 8.1 | |
| RandomModel=Qwen3-8B, beta=10%, err=.558, rho=.1782026.06 | 55.1 | — |