Code Generation on HumanEval (Normalized Spearman distance)
100Normalized Spearman DistanceBlock top-1
Evaluation results
| Method | Links | |
|---|---|---|
| Block top-1Perturbation Level=Task-prompt2026.06 | 100 | |
| SingleGraph1Perturbation Level=Task-prompt2026.06 | 96 | |
| Block top-2Perturbation Level=Task-prompt2026.06 | 95 | |
| ScoreWin80Perturbation Level=Task-prompt2026.06 | 93 | |
| NoTrunc bootPerturbation Level=Task-prompt2026.06 | 92 | |
| Random50→40×10Perturbation Level=Task-prompt2026.06 | 92 | |
| Trunc50→40Perturbation Level=Task-prompt2026.06 | 82 | |
| Block top-2Perturbation Level=Judge-prompt with Prometheus2026.06 | 74 | |
| Block top-1Perturbation Level=Judge-prompt with Prometheus2026.06 | 70 | |
| NoTrunc bootPerturbation Level=Judge-prompt with Prometheus2026.06 | 66 | |
| Random50→20×10Perturbation Level=Judge-prompt with Prometheus2026.06 | 66 | |
| ScoreWin40Perturbation Level=Judge-prompt with Prometheus2026.06 | 65 | |
| SingleGraph1/all10Perturbation Level=Judge-prompt with Prometheus2026.06 | 60 | |
| Trunc25→20Perturbation Level=Judge-prompt with Prometheus2026.06 | 57 |