Code Generation on HumanEval (Accuracy, Friedman Rank, P(EDEN > competitor))
31.3AccuracyEDEN
Evaluation results
| Method | Links | |||
|---|---|---|---|---|
| EDENtotal number of branches explored=9652026.05 | 31.3 | 1.25 | 0.75 | |
| Beam Searchbeam width=3, total number of branches explored=12002026.05 | 28.8 | 2.25 | 0.78 | |
| Top-k Sampling2026.05 | 27.6 | 7 | 1 | |
| Greedy2026.05 | 27 | 6.12 | 0.99 | |
| Top-p Sampling2026.05 | 27 | 6.62 | 0.99 | |
| Top-H Sampling2026.05 | 27 | 8.12 | 1 | |
| Best-of-nn=52026.05 | 27 | 4.62 | 0.96 | |
| Diverse Beam Searchbeam width=3, total number of branches explored=12002026.05 | 26.4 | 4.75 | 0.96 | |
| Min-p Sampling2026.05 | 25.8 | 8 | 1 | |
| Majority-nn=52026.05 | 19.6 | 6.25 | 0.99 |