Code Generation on BigCode
46.8AccuracyMCNIG
Evaluation results
| Method | Links | |
|---|---|---|
| MCNIGModel Size=8B2026.03 | 46.8 | |
| Self-CertaintyModel=Qwen3-14B, Number of Samples=322026.04 | 46.2 | |
| Lowest CentroidModel=Qwen3-14B, Number of Samples=322026.04 | 46 | |
| Tail ConfidenceModel=Qwen3-14B, Number of Samples=322026.04 | 45.4 | |
| Greedy DecodingModel=Olmo-3.1-Think, Number of Samples=322026.04 | 44.9 | |
| Lowest CentroidModel=Olmo-3.1-Think, Number of Samples=322026.04 | 44.5 | |
| Lowest CentroidModel=QWQ-32B, Number of Samples=322026.04 | 44.5 | |
| ORMModel Size=8B2026.03 | 44.1 | |
| Bottom WindowModel=QWQ-32B, Number of Samples=322026.04 | 44.1 | |
| Bottom WindowModel=Qwen3-14B, Number of Samples=322026.04 | 44 | |
| Self-CertaintyModel=QWQ-32B, Number of Samples=322026.04 | 44 | |
| Pass@1Model=QWQ-32B, Number of Samples=322026.04 | 43.9 | |
| Bottom WindowModel=Olmo-3.1-Think, Number of Samples=322026.04 | 43.4 | |
| Greedy DecodingModel=QWQ-32B, Number of Samples=322026.04 | 43.4 | |
| Self-CertaintyModel=Olmo-3.1-Think, Number of Samples=322026.04 | 43.3 | |
| IGModel Size=8B2026.03 | 43 | |
| Pass@1Model=Olmo-3.1-Think, Number of Samples=322026.04 | 42.6 | |
| Tail ConfidenceModel=QWQ-32B, Number of Samples=322026.04 | 42.3 | |
| Tail ConfidenceModel=Olmo-3.1-Think, Number of Samples=322026.04 | 41.5 | |
| Pass@1Model=Qwen3-14B, Number of Samples=322026.04 | 39.9 | |
| Greedy DecodingModel=Qwen3-14B, Number of Samples=322026.04 | 39.8 | |
| Tail ConfidenceModel=GPT-OSS-120B, Number of Samples=322026.04 | 36.8 | |
| Greedy DecodingModel=GPT-OSS-120B, Number of Samples=322026.04 | 36.1 | |
| Bottom WindowModel=GPT-OSS-120B, Number of Samples=322026.04 | 36.1 | |
| Lowest CentroidModel=GPT-OSS-120B, Number of Samples=322026.04 | 35.8 | |
| Pass@1Model=GPT-OSS-120B, Number of Samples=322026.04 | 35 | |
| OVM2026.03 | 34.9 | |
| Self-CertaintyModel=GPT-OSS-120B, Number of Samples=322026.04 | 34.9 | |
| QwenPRMModel Size=7B2026.03 | 31.5 | |
| Granite PRM v22026.03 | 31.4 | |
| MathShepherd2026.03 | 29.5 | |
| ImplicitPRM2026.03 | 29.3 | |
| Majority voting2026.03 | 27.6 | |
| Greedy DecodingModel=Ministral-3-14B-Instruct, Number of Samples=322026.04 | 26.3 | |
| Lowest CentroidModel=Ministral-3-14B-Instruct, Number of Samples=322026.04 | 25.5 | |
| Single sampling2026.03 | 23.3 | |
| Pass@1Model=Ministral-3-14B-Instruct, Number of Samples=322026.04 | 22.9 | |
| Tail ConfidenceModel=Ministral-3-14B-Instruct, Number of Samples=322026.04 | 19.1 | |
| Bottom WindowModel=Ministral-3-14B-Instruct, Number of Samples=322026.04 | 19 | |
| Self-CertaintyModel=Ministral-3-14B-Instruct, Number of Samples=322026.04 | 18 |