Code Debugging on HumanEval
96.3AccuracyMGDebugger
Evaluation results
| Method | Links | ||
|---|---|---|---|
| MGDebuggerModel=GPT-4o2024.10 | 96.3 | — | |
| MGDebuggerModel=Claude 3.5 Sonnet2024.10 | 95.7 | — | |
| ReflexionModel=GPT-4o2024.10 | 94.5 | — | |
| MGDebuggerBackbone=DeepSeek-Coder-V2-Lite2024.10 | 94.5 | 76.3 | |
| MGDebuggerBackbone=Codestral2024.10 | 94.5 | 77.5 | |
| ReflexionModel=Claude 3.5 Sonnet2024.10 | 93.9 | — | |
| Self-Debugging (Expl.)Model=GPT-4o2024.10 | 93.3 | — | |
| Self-Debugging (Expl.)Model=Claude 3.5 Sonnet2024.10 | 93.3 | — | |
| MGDebuggerModel=LLaMA 3.1 70B2024.10 | 92.7 | — | |
| MGDebuggerBackbone=CodeQwen1.52024.10 | 91.5 | 64.1 | |
| ReflexionBackbone=DeepSeek-Coder-V2-Lite2024.10 | 90.9 | 60.5 | |
| No-DebuggingModel=GPT-4o2024.10 | 90.2 | — | |
| No-DebuggingModel=Claude 3.5 Sonnet2024.10 | 89.6 | — | |
| ReflexionModel=LLaMA 3.1 70B2024.10 | 89.6 | — | |
| Self-Debugging (Expl.)Backbone=Codestral2024.10 | 89.6 | 57.5 | |
| Simple FeedbackBackbone=Codestral2024.10 | 88.4 | 52.5 | |
| Self-Debugging (Expl.)Backbone=CodeQwen1.52024.10 | 87.8 | 48.7 | |
| ReflexionBackbone=CodeQwen1.52024.10 | 87.8 | 48.7 | |
| Self-Debugging (Expl.)Backbone=DeepSeek-Coder-V2-Lite2024.10 | 87.2 | 44.7 | |
| Self-Debugging (Expl.)Model=LLaMA 3.1 70B2024.10 | 86.6 | — | |
| ReflexionBackbone=Codestral2024.10 | 86.6 | 45 | |
| Self-Debugging (Trace)Backbone=DeepSeek-Coder-V2-Lite2024.10 | 86 | 39.5 | |
| Self-EditBackbone=Codestral2024.10 | 86 | 42.5 | |
| Simple FeedbackBackbone=CodeQwen1.52024.10 | 85.4 | 38.5 | |
| Self-Debugging (Trace)Backbone=CodeQwen1.52024.10 | 84.8 | 35.9 | |
| LDB (Block)Backbone=DeepSeek-Coder-V2-Lite2024.10 | 84.1 | 31.6 | |
| Self-EditBackbone=CodeQwen1.52024.10 | 84.1 | 33.3 | |
| Self-Debugging (Trace)Backbone=Codestral2024.10 | 84.1 | 35 | |
| LDB (Block)Backbone=Codestral2024.10 | 83.5 | 32.5 | |
| LDB (Line)Backbone=Codestral2024.10 | 83.5 | 32.5 | |
| Self-EditBackbone=DeepSeek-Coder-V2-Lite2024.10 | 82.9 | 26.3 | |
| Simple FeedbackBackbone=DeepSeek-Coder-V2-Lite2024.10 | 82.3 | 23.7 | |
| LDB (Line)Backbone=DeepSeek-Coder-V2-Lite2024.10 | 82.3 | 23.7 | |
| LDB (Function)Backbone=Codestral2024.10 | 82.3 | 27.5 | |
| LDB (Function)Backbone=DeepSeek-Coder-V2-Lite2024.10 | 81.7 | 21.1 | |
| LDB (Function)Backbone=CodeQwen1.52024.10 | 80.5 | 17.9 | |
| LDB (Line)Backbone=CodeQwen1.52024.10 | 79.9 | 15.4 | |
| No-DebuggingModel=LLaMA 3.1 70B2024.10 | 79.3 | — | |
| LDB (Block)Backbone=CodeQwen1.52024.10 | 79.3 | 12.8 | |
| No-DebuggingBackbone=DeepSeek-Coder-V2-Lite2024.10 | 76.8 | — | |
| No-DebuggingBackbone=CodeQwen1.52024.10 | 76.2 | — | |
| No-DebuggingBackbone=Codestral2024.10 | 75.6 | — |