Code Generation on TransCoder (test)
87.7Pass@1 AccLDB
Evaluation results
| Method | Links | ||
|---|---|---|---|
| LDBModel=GPT-3.5 (>=175B), Debugger=LDB2024.02 | 87.7 | 5.4 | |
| SD (+Trace)Model=GPT-3.5 (>=175B), Debugger=SD (+Trace)2024.02 | 86.1 | 3.8 | |
| SD (+Expl.)Model=GPT-3.5 (>=175B), Debugger=SD (+Expl.)2024.02 | 85.9 | 3.6 | |
| BaselineModel=GPT-3.5 (>=175B), Debugger=None2024.02 | 82.3 | — | |
| LDBModel=CodeLlama (34B), Debugger=LDB2024.02 | 79.6 | 9.8 | |
| SD (+Expl.)Model=CodeLlama (34B), Debugger=SD (+Expl.)2024.02 | 79.4 | 9.6 | |
| SD (+Trace)Model=CodeLlama (34B), Debugger=SD (+Trace)2024.02 | 76.4 | 6.6 | |
| BaselineModel=CodeLlama (34B), Debugger=None2024.02 | 69.8 | — | |
| LDBModel=StarCoder (15B), Debugger=LDB2024.02 | 69.8 | 8 | |
| SD (+Expl.)Model=StarCoder (15B), Debugger=SD (+Expl.)2024.02 | 68.9 | 7.1 | |
| SD (+Trace)Model=StarCoder (15B), Debugger=SD (+Trace)2024.02 | 65.7 | 3.9 | |
| BaselineModel=StarCoder (15B), Debugger=None2024.02 | 61.8 | — |