Code Generation on MBPP (Pass%, CodeBLEU)
99.78Pass%Iterative
Evaluation results
| Method | Links | ||||||
|---|---|---|---|---|---|---|---|
| IterativeModels=Gemini2026.06 | 99.78 | 1.2 | 0.974 | 70.7 | 89.1 | 0.46 | |
| Refactoring-BasedModels=Claude2026.06 | 99.67 | — | 0.723 | 8 | 8 | 0.65 | |
| Refactoring-BasedModels=Gemini2026.06 | 99.45 | — | 0.879 | 11.1 | 42.2 | 0.91 | |
| StaticModels=Gemini2026.06 | 98.25 | — | 0.922 | 39.8 | 59.7 | 0.44 | |
| IterativeModels=Claude2026.06 | 97.27 | 1.33 | 0.989 | 90 | 96 | 0.46 | |
| Iterative-AGLModels=Gemini2026.06 | 96.79 | 1.9 | 0.994 | 96.7 | 97.9 | 0.46 | |
| Comment-basedModels=Gemini2026.06 | 95.01 | — | 0.925 | 41.8 | 68.6 | 0.44 | |
| ICWModels=Gemini2026.06 | 94.44 | — | 0.512 | 4.5 | 7.2 | 0.44 | |
| Iterative-AGLModels=Claude2026.06 | 93.6 | 2.5 | 0.987 | 87 | 94 | 0.47 | |
| ICWModels=Claude2026.06 | 92.4 | — | 0.77 | 2 | 14 | 0.5 | |
| Comment-basedModels=Claude2026.06 | 90 | — | 0.892 | 39 | 60 | 0.47 | |
| StaticModels=Claude2026.06 | 86.73 | — | 0.899 | 26 | 56 | 0.46 | |
| No WatermarkModels=Gemini2026.06 | 64.77 | — | 0.577 | 3.5 | 10.6 | 0.42 | |
| No WatermarkModels=Claude2026.06 | 63.96 | — | 0.416 | 4 | 6 | 0.5 | |
| SWEETModels=Qwen2026.06 | 43.83 | — | 0.92 | 44 | 72 | 0.42 |