Program Repair on Real misspecified programs neutral briefs
100Repair Success RateGemini-2.5-flash
Evaluation results
| Method | Links | |
|---|---|---|
| Gemini-2.5-flashFeedback regime=judge2026.06 | 100 | |
| Gemini-2.5-flashFeedback regime=diag2026.06 | 100 | |
| Claude 4.6Feedback regime=judge2026.06 | 100 | |
| Claude 4.6Feedback regime=diag2026.06 | 100 | |
| Gemini-3.1-proFeedback regime=none2026.06 | 100 | |
| Gemini-3.1-proFeedback regime=summ.2026.06 | 100 | |
| GPT-4.1Feedback regime=diag2026.06 | 93 | |
| GPT-5.1Feedback regime=diag2026.06 | 92 | |
| DeepSeek-V4-flashFeedback regime=diag2026.06 | 90 | |
| pooled‡Feedback regime=diag2026.06 | 84 | |
| GPT-5.1Feedback regime=check.2026.06 | 77 | |
| Gemini-2.5-proFeedback regime=test2026.06 | 75 | |
| Gemini-2.5-proFeedback regime=summ.2026.06 | 75 | |
| Gemini-2.5-proFeedback regime=check.2026.06 | 75 | |
| Gemini-2.5-proFeedback regime=judge2026.06 | 75 | |
| Gemini-2.5-proFeedback regime=diag2026.06 | 75 | |
| Gemini-3.1-proFeedback regime=judge2026.06 | 71 | |
| Gemini-3.1-proFeedback regime=diag2026.06 | 71 | |
| pooled‡Feedback regime=judge2026.06 | 71 | |
| Gemini-2.5-flashFeedback regime=summ.2026.06 | 67 | |
| Gemini-2.5-flashFeedback regime=check.2026.06 | 67 | |
| DeepSeek-V4-proFeedback regime=judge2026.06 | 67 | |
| DeepSeek-V4-proFeedback regime=diag2026.06 | 67 | |
| GPT-4.1Feedback regime=judge2026.06 | 64 | |
| pooled‡Feedback regime=check.2026.06 | 63 | |
| DeepSeek-V4-flashFeedback regime=check.2026.06 | 60 | |
| DeepSeek-V4-flashFeedback regime=judge2026.06 | 60 | |
| GPT-4.1Feedback regime=check.2026.06 | 57 | |
| Gemini-3.1-proFeedback regime=check.2026.06 | 57 | |
| GPT-5.1Feedback regime=none2026.06 | 54 | |
| Claude 4.6Feedback regime=none2026.06 | 50 | |
| Claude 4.6Feedback regime=summ.2026.06 | 50 | |
| Claude 4.6Feedback regime=check.2026.06 | 50 | |
| DeepSeek-V4-flashFeedback regime=none2026.06 | 50 | |
| Gemini-2.5-proFeedback regime=none2026.06 | 50 | |
| GPT-5.1Feedback regime=test2026.06 | 46 | |
| GPT-5.1Feedback regime=summ.2026.06 | 46 | |
| GPT-5.1Feedback regime=judge2026.06 | 46 | |
| pooled‡Feedback regime=none2026.06 | 46 | |
| pooled‡Feedback regime=summ.2026.06 | 46 | |
| DeepSeek-V4-proFeedback regime=none2026.06 | 44 | |
| GPT-4o-miniFeedback regime=none2026.06 | 41 | |
| GPT-4.1Feedback regime=none2026.06 | 36 | |
| pooled‡Feedback regime=test2026.06 | 36 | |
| Gemini-2.5-flashFeedback regime=none2026.06 | 33 | |
| Gemini-2.5-flashFeedback regime=test2026.06 | 33 | |
| DeepSeek-V4-proFeedback regime=check.2026.06 | 33 | |
| DeepSeek-V4-flashFeedback regime=summ.2026.06 | 30 | |
| GPT-4o-miniFeedback regime=diag2026.06 | 18 | |
| GPT-4.1Feedback regime=summ.2026.06 | 14 | |
| GPT-4o-miniFeedback regime=test2026.06 | 12 | |
| GPT-4o-miniFeedback regime=judge2026.06 | 12 | |
| DeepSeek-V4-flashFeedback regime=test2026.06 | 10 | |
| GPT-4o-miniFeedback regime=summ.2026.06 | 0 | |
| GPT-4o-miniFeedback regime=check.2026.06 | 0 | |
| GPT-4.1Feedback regime=test2026.06 | 0 | |
| Claude 4.6Feedback regime=test2026.06 | 0 | |
| DeepSeek-V4-proFeedback regime=test2026.06 | 0 | |
| DeepSeek-V4-proFeedback regime=summ.2026.06 | 0 | |
| Gemini-3.1-proFeedback regime=test2026.06 | 0 |