Code Generation on HumanEval-NFR
25.19Pass@1ARCHCODE
Evaluation Results
| Method | Links | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| ARCHCODE2024.08 | 25.19 | 27.33 | 62.86 | 69.7 | 40.86 | 42.72 | 56.43 | 62.23 | 68.53 | 74.67 | |
| GPT-3.5-TurboCoT=true2024.08 | 5 | 12.08 | 50 | 66.03 | 7.32 | 17.67 | 44.33 | 62 | 45.49 | 66.83 | |
| CODETCoT=false2024.08 | 3.03 | 10.03 | 58.5 | 67.31 | 4.61 | 14.52 | 57.8 | 68.5 | 47.62 | 74.9 | |
| GPT-3.5-TurboCoT=false2024.08 | 2.62 | 10.03 | 53.48 | 65.75 | 4.21 | 14.55 | 53.23 | 68.38 | 20.98 | 36.72 |