Agentic Programming on SWE-bench-derived OOD (test)
75.89Average Performance (%)Oracle (cost-aware)
Evaluation results
| Method | Links | ||||
|---|---|---|---|---|---|
| Oracle (cost-aware)Family=Bound2026.06 | 75.89 | 0 | 32.71 | 2.32 | |
| ACRouterFamily=Agentic2026.06 | 62.5 | 17 | 52.97 | 1.18 | |
| Always-Opus 4.6Family=Single-Model2026.06 | 57.14 | 26.7 | 89.28 | 0.64 | |
| Qwen3.5-0.8B-FinetunedFamily=Trained2026.06 | 55.36 | 27.2 | 74.81 | 0.74 | |
| LinUCBFamily=Bandit, Aggregation=5-seed avg2026.06 | 49.82 | 31.1 | 51.9 | 0.96 | |
| LinTSFamily=Bandit, Aggregation=5-seed avg2026.06 | 46.43 | 35.9 | 61.91 | 0.75 | |
| RandomFamily=Single-Model2026.06 | 31.25 | 50.4 | 36.76 | 0.85 | |
| RouteLLM-BERTFamily=Trained2026.06 | 21.43 | 59.4 | 16.48 | 1.3 | |
| LogRegFamily=Trained2026.06 | 19.64 | 61.8 | 16.79 | 1.17 | |
| Always-Kimi-K2.5Family=Single-Model2026.06 | 18.75 | 62.3 | 15.37 | 1.22 | |
| kNN RetrievalFamily=Heuristic2026.06 | 14.29 | 66.7 | 9.86 | 1.45 | |
| TF-IDF+MLPFamily=Trained2026.06 | 13.39 | 67.9 | 11.44 | 1.17 | |
| RouteLLM-MFFamily=Trained2026.06 | 8.93 | 72.7 | 9.5 | 0.94 | |
| Always-Qwen3.5-PlusFamily=Single-Model2026.06 | 2.68 | 80.1 | 14.11 | 0.19 |