Merge Conflict Resolution on Merge-Bench Java (test)
54.7Equivalent Text RateGemini 2.5 Pro
Evaluation results
| Method | Links | |||||
|---|---|---|---|---|---|---|
| Gemini 2.5 Pro2026.05 | 54.7 | 62.5 | 34.1 | 3.4 | 0 | |
| LLMergeJParameters=14B2026.05 | 48.8 | 58.9 | 35.5 | 5.6 | 0 | |
| o3 Pro2026.05 | 46.1 | 54.3 | 35.1 | 10.6 | 0 | |
| Claude Opus 42026.05 | 44.4 | 51.2 | 27.6 | 21.2 | 0 | |
| Qwen3Parameters=14B, Training=SFT2026.05 | 36.6 | 44.7 | 36.1 | 19.2 | 0 | |
| R1-0528Parameters=671B2026.05 | 35.9 | 42.4 | 24 | 33.2 | 0.4 | |
| Grok 42026.05 | 33.4 | 39.7 | 17.9 | 42.4 | 0 | |
| QwQParameters=32B2026.05 | 32.1 | 43.2 | 30.9 | 20.5 | 5.5 | |
| Qwen3Parameters=235B2026.05 | 30.9 | 39.5 | 25.4 | 35.1 | 0 | |
| Llama 4 Maverick2026.05 | 26.2 | 32.6 | 27.1 | 40.1 | 0.2 | |
| R1Parameters=70B2026.05 | 25.7 | 33 | 26.9 | 39.6 | 0.5 | |
| R1Parameters=32B2026.05 | 22.8 | 30.4 | 29.3 | 39.7 | 0.6 | |
| Qwen3Parameters=32B2026.05 | 13.2 | 16.9 | 11.1 | 71.8 | 0.1 | |
| Qwen3Parameters=14B2026.05 | 12.9 | 16.6 | 8.7 | 74.7 | 0 | |
| R1Parameters=14B2026.05 | 9.3 | 13.4 | 15.4 | 70.7 | 0.5 | |
| Qwen3Parameters=8B2026.05 | 5.5 | 9.1 | 4.7 | 86.1 | 0.1 | |
| R1Parameters=8B2026.05 | 3.5 | 8.1 | 31.6 | 58.4 | 1.9 | |
| R1Parameters=1.5B2026.05 | 0 | 0.2 | 46.5 | 44 | 9.3 |