LLM Routing on SWE-Bench Verified n=500
92.1QR (%)OpenRouter Auto
Evaluation results
| Method | Links | |||
|---|---|---|---|---|
| OpenRouter AutoModel Pool (strong → cheap)=Claude Sonnet 4.6, GPT-5.2, GPT-5 mini, Mode / op. point=default auto2026.05 | 92.1 | 0.1 | 76 | |
| RouteLLM (BERT)Model Pool (strong → cheap)=GPT-5.3 Codex, GPT-5.4 mini, Mode / op. point=threshold = 0.52026.05 | 91.1 | 18.6 | 45.2 | |
| HyDRA (cons.)Model Pool (strong → cheap)=Claude Sonnet 4.6, Claude Haiku 4.5, GPT-5.4, GPT-5.3 Codex, GPT-5.4 mini, Mode / op. point=τ=0.2392026.05 | 86.1 | 54.1 | 70.4 | |
| HyDRA (agg.)Model Pool (strong → cheap)=Claude Sonnet 4.6, Claude Haiku 4.5, GPT-5.4, GPT-5.3 Codex, GPT-5.4 mini, Mode / op. point=τ=0.4362026.05 | 84.5 | 63.7 | 52.8 | |
| Avengers ProModel Pool (strong → cheap)=Claude Sonnet 4.6, Claude Haiku 4.5, GPT-5.4, GPT-5.3 Codex, GPT-5.4 mini, Mode / op. point=default2026.05 | 83.3 | 51 | 50.8 | |
| Azure Foundry RouterModel Pool (strong → cheap)=GPT-5.2, GPT-5, GPT-5 mini, Mode / op. point=Balanced2026.05 | 76 | 66.2 | 17.6 |