Code Generation on BigCodeBench (hold-out)
35.8Pass@1Claude-4-Sonnet
Evaluation results
| Method | Links | |
|---|---|---|
| Claude-4-SonnetZero-shot=true2025.12 | 35.8 | |
| TRINITYZero-shot=true2025.12 | 35.8 | |
| Gemini Pro 2.5Zero-shot=true2025.12 | 35.1 | |
| GPT-5Zero-shot=true2025.12 | 33.8 | |
| DeepSeek-R1-Qwen-32BZero-shot=true2025.12 | 24.3 | |
| Qwen3-32BZero-shot=true, Evaluation Mode=direct2025.12 | 23 | |
| Qwen3-32BZero-shot=true, Evaluation Mode=reasoning2025.12 | 20.9 | |
| Gemma-3-27B-ITZero-shot=true2025.12 | 20.3 |