Code Generation on Code Benchmarks (HumanEval, MBPP)
30.1HumanEval ScoreDataAgentSFT
Evaluation results
| Method | Links | ||||
|---|---|---|---|---|---|
| DataAgentSFTModel Architecture=Pythia-1.4B2025.07 | 30.1 | 29.1 | 25.35 | — | |
| Naive TrainingGPU Hrs=3390.772025.07 | 27.3 | 37.8 | 32.55 | 40.08 | |
| Naive TrainingModel Architecture=LLaMA-DCLM2025.07 | 27.3 | 37.8 | 32.55 | — | |
| Naive TrainingModel Architecture=Pythia-1.4B2025.07 | 24.7 | 32.4 | 28.55 | — | |
| DataAgentRLModel Architecture=Pythia-1.4B2025.07 | 23.3 | 30 | 26.65 | — | |
| DataAgentSFTGPU Hrs=5318.372025.07 | 22.4 | 32.8 | 27.6 | 45.07 | |
| DataAgentSFTModel Architecture=LLaMA-DCLM2025.07 | 22.4 | 32.8 | 27.6 | — | |
| DataAgentRLGPU Hrs=5461.822025.07 | 22 | 34.1 | 28.05 | 46.3 | |
| DataAgentRLModel Architecture=LLaMA-DCLM2025.07 | 22 | 34.1 | 28.05 | — | |
| RegMixGPU Hrs=5726.112025.07 | 21.1 | 31.6 | 26.35 | 44.85 | |
| RegMixModel Architecture=LLaMA-DCLM2025.07 | 21.1 | 31.6 | 26.35 | — | |
| RegMixModel Architecture=Pythia-1.4B2025.07 | 20.7 | 27 | 23.85 | — | |
| DBLGPU Hrs=5480.742025.07 | 20.6 | 30.3 | 25.45 | 43.52 | |
| DBLModel Architecture=LLaMA-DCLM2025.07 | 20.6 | 30.3 | 25.45 | — | |
| Base ModelGPU Hrs=–2025.07 | 8.6 | 14.2 | 11.4 | 44.52 | |
| Base ModelModel Architecture=LLaMA-DCLM2025.07 | 8.6 | 14.2 | 11.4 | — | |
| Base ModelModel Architecture=Pythia-1.4B2025.07 | 4.9 | 4.9 | 4.9 | — |