Coding on HumanEval (Accuracy, Delta, Pct. gap closed (%))
93.9AccuracyOracle
Evaluation results
| Method | Links | |||
|---|---|---|---|---|
| OracleModel=GPT-OSS-20B, Calibration set=KodCode2026.04 | 93.9 | — | — | |
| SCATRModel=GPT-OSS-20B, Calibration set=KodCode2026.04 | 93.2 | 1.5 | 68.2 | |
| OracleModel=Qwen-30B-A3B, Calibration set=KodCode2026.04 | 92.7 | — | — | |
| LoRAModel=GPT-OSS-20B, Calibration set=KodCode2026.04 | 92.3 | — | — | |
| Best Confidence-based StrategyModel=GPT-OSS-20B, Calibration set=KodCode2026.04 | 91.7 | — | — | |
| SCATRModel=Qwen-30B-A3B, Calibration set=KodCode2026.04 | 90.4 | 0.4 | 14.8 | |
| Best Confidence-based StrategyModel=Qwen-30B-A3B, Calibration set=KodCode2026.04 | 90 | — | — | |
| PRMModel=GPT-OSS-20B, Calibration set=KodCode2026.04 | 89.8 | — | — | |
| PRMModel=Qwen-30B-A3B, Calibration set=KodCode2026.04 | 87.6 | — | — | |
| OracleModel=Qwen-1.7B, Calibration set=KodCode2026.04 | 87.4 | — | — | |
| OracleModel=OLMo-2-7B, Calibration set=KodCode2026.04 | 73.4 | — | — | |
| LoRAModel=Qwen-1.7B, Calibration set=KodCode2026.04 | 63.8 | — | — | |
| PRMModel=Qwen-1.7B, Calibration set=KodCode2026.04 | 60.4 | — | — | |
| SCATRModel=Qwen-1.7B, Calibration set=KodCode2026.04 | 59.3 | 3.2 | 10.2 | |
| Best Confidence-based StrategyModel=Qwen-1.7B, Calibration set=KodCode2026.04 | 56.1 | — | — | |
| LoRAModel=OLMo-2-7B, Calibration set=KodCode2026.04 | 37.7 | — | — | |
| SCATRModel=OLMo-2-7B, Calibration set=KodCode2026.04 | 35 | 3.3 | 7.9 | |
| PRMModel=OLMo-2-7B, Calibration set=KodCode2026.04 | 32.9 | — | — | |
| Best Confidence-based StrategyModel=OLMo-2-7B, Calibration set=KodCode2026.04 | 31.7 | — | — |