LLM Routing on SWE-bench Lite N=300 (test)
179.75LossPriority-aware (oracle; upper bound)
Evaluation Results
| Method | Links | |||
|---|---|---|---|---|
| Priority-aware (oracle; upper bound)Strategy=Priority-aware, Predictor=oracle, Compute budget=top-25% premium routing, Price ratio=1:42026.06 | 179.75 | 33 | 27.8 | |
| Priority-aware (Qwen predictor; primary deployable variant)Strategy=Priority-aware, Predictor=Qwen, Compute budget=top-25% premium routing, Price ratio=1:42026.06 | 186 | 30.7 | 27.4 | |
| Consequence-aware (oracle)Strategy=Consequence-aware, Predictor=oracle, Compute budget=top-25% premium routing, Price ratio=1:42026.06 | 205.75 | 23.3 | 18.7 | |
| Consequence-aware (Qwen predictor; deployable)Strategy=Consequence-aware, Predictor=Qwen, Compute budget=top-25% premium routing, Price ratio=1:42026.06 | 209.75 | 21.8 | 18.4 | |
| Consequence-aware (Claude predictor; cross-model robustness)Strategy=Consequence-aware, Predictor=Claude, Compute budget=top-25% premium routing, Price ratio=1:42026.06 | 220.5 | 17.8 | 17.4 | |
| RandomStrategy=Random, Compute budget=top-25% premium routing, Price ratio=1:42026.06 | 233 | 13.1 | 17 | |
| Difficulty-aware (Snell-style)Strategy=Difficulty-aware (Snell-style), Compute budget=top-25% premium routing, Price ratio=1:42026.06 | 268.25 | 0 | 5.4 |