Code Generation on HumanEval (Pass@1, Avg Acc, Avg TPF)
87.2Pass@1Nemotron-Labs-Diffusion-14B
Evaluation results
| Method | Links | |||
|---|---|---|---|---|
| Nemotron-Labs-Diffusion-14BScale=14B, Gen. Mode=Quad. SS2026.07 | 87.2 | 68.15 | 6.92 | |
| Nemotron-Labs-Diffusion-14BScale=14B, Gen. Mode=AR2026.07 | 86.59 | 67.46 | 1 | |
| Nemotron-Labs-Diffusion-14BScale=14B, Gen. Mode=Linear SS2026.07 | 85.37 | 66.36 | 5.96 | |
| Phi-4-14BScale=14B, Gen. Mode=AR2026.07 | 84.6 | 60.82 | 1 | |
| Qwen3-14BScale=14B, Gen. Mode=AR2026.07 | 83.54 | 65.17 | 1 | |
| Nemotron-Labs-Diffusion-14BScale=14B, Gen. Mode=Diff.2026.07 | 83.54 | 66.51 | 2.74 | |
| Nemotron-Labs-Diffusion-3BScale=3B, Gen. Mode=Quad. SS2026.07 | 79.27 | 55.8 | 5.42 | |
| Phi-4-mini-InstructScale=3B, Gen. Mode=AR2026.07 | 77.44 | 49.2 | 1 | |
| Nemotron-Labs-Diffusion-3BScale=3B, Gen. Mode=AR2026.07 | 76.22 | 55.5 | 1 | |
| Qwen3-4BScale=3B, Gen. Mode=AR2026.07 | 75.91 | 53.23 | 1 | |
| Nemotron-Labs-Diffusion-3BScale=3B, Gen. Mode=Linear SS2026.07 | 75 | 55 | 4.36 | |
| Nemotron-Labs-Diffusion-3BScale=3B, Gen. Mode=Diff.2026.07 | 74.39 | 52.9 | 1.91 | |
| Ministral3-14B-Instruct-2512Scale=14B, Gen. Mode=AR2026.07 | 72.56 | 62.97 | 1 | |
| Phi-3-Medium-14BScale=14B, Gen. Mode=AR2026.07 | 70.43 | 49.34 | 1 | |
| Llama-3.2-3B-InstructScale=3B, Gen. Mode=AR2026.07 | 65.85 | 39.9 | 1 | |
| Ministral3-3B-Instruct-2512Scale=3B, Gen. Mode=AR2026.07 | 64.79 | 47.97 | 1 | |
| Gemma-3-12B-ITScale=14B, Gen. Mode=AR2026.07 | 58.23 | 58.03 | 1 |