Code Generation on BigCodeBench (Accuracy, Average tokens generated)
48.51AccuracyGemma-4 31B
Evaluation results
| Method | Links | ||
|---|---|---|---|
| Gemma-4 31BEvaluation Mode=Reasoning2026.06 | 48.51 | 2,198 | |
| Gemma-4 31BEvaluation Mode=Reasoning+Rotate2Think2026.06 | 48.25 | 2,291 | |
| Gemma-4 31BEvaluation Mode=Base+Rotate2Think2026.06 | 47.54 | 1,577 | |
| Gemma-4 31BEvaluation Mode=Base2026.06 | 46.49 | 364 | |
| Phi-4 14BEvaluation Mode=Reasoning+Rotate2Think2026.06 | 44.03 | 3,962 | |
| Phi-4 14BEvaluation Mode=Reasoning2026.06 | 43.68 | 4,172 | |
| Qwen3 4BEvaluation Mode=Reasoning+Rotate2Think2026.06 | 41.84 | 5,670 | |
| Qwen3 4BEvaluation Mode=Reasoning2026.06 | 41.58 | 5,787 | |
| Phi-4 14BEvaluation Mode=Base2026.06 | 39.91 | 242 | |
| Phi-4 14BEvaluation Mode=Base+Rotate2Think2026.06 | 39.74 | 582 | |
| Qwen3 4BEvaluation Mode=Base+Rotate2Think2026.06 | 38.51 | 404 | |
| Gemma-4 E4BEvaluation Mode=Base+Rotate2Think2026.06 | 38.38 | 1,238 | |
| Qwen3 4BEvaluation Mode=Base2026.06 | 38.24 | 409 | |
| Gemma-4 E4BEvaluation Mode=Reasoning+Rotate2Think2026.06 | 37.98 | 1,252 | |
| Gemma-4 E4BEvaluation Mode=Reasoning2026.06 | 36.84 | 1,227 | |
| Gemma-4 E4BEvaluation Mode=Base2026.06 | 36.67 | 608 |