Code Generation on CodeContests (Pass@1)
55.2Pass@1Oracle
Evaluation Results
| Method | Links | |
|---|---|---|
| OracleModels=182026.05 | 55.2 | |
| MARGIN c.Models=18, Confidence Source=consistency2026.05 | 45.7 | |
| CAPSModel=Qwen3-4B-Thinking-2507, N=16, Temperature=0.62026.05 | 44.8 | |
| MARGIN v.Models=18, Confidence Source=verbalized2026.05 | 40.7 | |
| V1-InferModel=Qwen3-4B-Thinking-2507, N=16, Temperature=0.62026.05 | 40.6 | |
| Best modelModels=182026.05 | 40 | |
| CAPSModel=Qwen3-14B, N=16, Temperature=0.62026.05 | 35.8 | |
| PointwiseModel=Qwen3-4B-Thinking-2507, N=16, Temperature=0.62026.05 | 34.1 | |
| VanillaModel=Qwen3-4B-Thinking-2507, N=16, Temperature=0.62026.05 | 31.4 | |
| V1-InferModel=Qwen3-14B, N=16, Temperature=0.62026.05 | 30.9 | |
| V1-InferModel=GPT-OSS-20B, N=16, Temperature=0.62026.05 | 30.9 | |
| CAPSModel=GPT-OSS-20B, N=16, Temperature=0.62026.05 | 29.4 | |
| Raw verb.Models=182026.05 | 27.3 | |
| PointwiseModel=Qwen3-14B, N=16, Temperature=0.62026.05 | 24.5 | |
| PointwiseModel=GPT-OSS-20B, N=16, Temperature=0.62026.05 | 24.5 | |
| VanillaModel=Qwen3-14B, N=16, Temperature=0.62026.05 | 21.7 | |
| VanillaModel=GPT-OSS-20B, N=16, Temperature=0.62026.05 | 21.4 | |
| CAPSModel=Qwen3-4B-Instruct-2507, N=16, Temperature=0.62026.05 | 14.5 | |
| V1-InferModel=Qwen3-4B-Instruct-2507, N=16, Temperature=0.62026.05 | 7.9 | |
| PointwiseModel=Qwen3-4B-Instruct-2507, N=16, Temperature=0.62026.05 | 4.5 | |
| VanillaModel=Qwen3-4B-Instruct-2507, N=16, Temperature=0.62026.05 | 1.3 |