Coding on KodCode
94.5AccuracyOracle
Evaluation results
| Method | Links | |||
|---|---|---|---|---|
| OracleModel=GPT-OSS-20B, Calibration set=HumanEval2026.04 | 94.5 | — | — | |
| OracleModel=Qwen-30B-A3B, Calibration set=HumanEval2026.04 | 93.7 | — | — | |
| OracleModel=Qwen-1.7B, Calibration set=HumanEval2026.04 | 90.8 | — | — | |
| SCATRModel=Qwen-30B-A3B, Calibration set=HumanEval2026.04 | 88.2 | 1.2 | 17.9 | |
| Best Confidence-based StrategyModel=Qwen-30B-A3B, Calibration set=HumanEval2026.04 | 87 | — | — | |
| SCATRModel=GPT-OSS-20B, Calibration set=HumanEval2026.04 | 86.8 | 3 | 28 | |
| LoRAModel=GPT-OSS-20B, Calibration set=HumanEval2026.04 | 85.2 | — | — | |
| PRMModel=Qwen-30B-A3B, Calibration set=HumanEval2026.04 | 84.6 | — | — | |
| Best Confidence-based StrategyModel=GPT-OSS-20B, Calibration set=HumanEval2026.04 | 83.8 | — | — | |
| PRMModel=GPT-OSS-20B, Calibration set=HumanEval2026.04 | 82.5 | — | — | |
| LoRAModel=Qwen-1.7B, Calibration set=HumanEval2026.04 | 68.3 | — | — | |
| SCATRModel=Qwen-1.7B, Calibration set=HumanEval2026.04 | 67.1 | 5.8 | 19.7 | |
| OracleModel=OLMo-2-7B, Calibration set=HumanEval2026.04 | 66.7 | — | — | |
| PRMModel=Qwen-1.7B, Calibration set=HumanEval2026.04 | 64.5 | — | — | |
| Best Confidence-based StrategyModel=Qwen-1.7B, Calibration set=HumanEval2026.04 | 61.3 | — | — | |
| SCATRModel=OLMo-2-7B, Calibration set=HumanEval2026.04 | 33.1 | 6.1 | 15.4 | |
| LoRAModel=OLMo-2-7B, Calibration set=HumanEval2026.04 | 32.7 | — | — | |
| PRMModel=OLMo-2-7B, Calibration set=HumanEval2026.04 | 29.3 | — | — | |
| Best Confidence-based StrategyModel=OLMo-2-7B, Calibration set=HumanEval2026.04 | 27 | — | — |