Model Routing for Coding Tasks on Agent-as-a-Router In-distribution (test)
57Average Performance (%)Oracle (cost-aware)
Evaluation results
| Method | Links | ||||
|---|---|---|---|---|---|
| Oracle (cost-aware)Family=Bound2026.06 | 57 | 0 | 6.95 | 8.2 | |
| ACRouterFamily=Agentic2026.06 | 49.98 | 205.5 | 13.21 | 3.79 | |
| DimensionBestFamily=Heuristic2026.06 | 47.5 | 277.4 | 12.89 | 3.69 | |
| LogRegFamily=Trained2026.06 | 47.26 | 284.4 | 7.54 | 6.27 | |
| RouteLLM-BERTFamily=Trained2026.06 | 47.22 | 285.5 | 7.59 | 6.22 | |
| kNN RetrievalFamily=Heuristic2026.06 | 47.18 | 286.7 | 7.77 | 6.07 | |
| TF-IDF+MLPFamily=Trained2026.06 | 46.97 | 292.8 | 7.69 | 6.11 | |
| LinUCB (5-seed avg)Family=Bandit2026.06 | 46.84 | 296.9 | 10.69 | 4.38 | |
| LinTS (5-seed avg)Family=Bandit2026.06 | 46.48 | 307.4 | 10.35 | 4.49 | |
| Qwen3.5-0.8B-FinetunedFamily=Trained2026.06 | 46.41 | 309.1 | 6.81 | 6.82 | |
| RouteLLM-MFFamily=Trained2026.06 | 46.16 | 316.5 | 7.46 | 6.19 | |
| Always-Opus 4.6Family=Single-Model2026.06 | 43.83 | 387.1 | 34.02 | 1.29 | |
| RandomFamily=Single-Model2026.06 | 38.75 | 533.6 | 15.64 | 2.48 | |
| Always-Qwen3.5-PlusFamily=Single-Model2026.06 | 37.16 | 580.2 | 18.14 | 2.05 | |
| Always-Kimi-K2.5Family=Single-Model2026.06 | 36.66 | 593.3 | 2.9 | 12.62 |