Query Routing and Tool-Calling Evaluation on Held-out 12,282 examples (test)
89.39AccuracyOracle
Evaluation results
| Method | Links | ||
|---|---|---|---|
| OracleType=Upper Bound2026.05 | 89.39 | 0.001 | |
| ModernBERT (149M)Type=Ours (Switchcraft), Backbone=ModernBERT, Parameters=149M2026.05 | 83.02 | 0.0006 | |
| DistilBERT (66M)Type=Ours (Switchcraft), Backbone=DistilBERT, Parameters=66M2026.05 | 82.94 | 0.0007 | |
| DeBERTa-v3 (86M)Type=Ours (Switchcraft), Backbone=DeBERTa-v3, Parameters=86M2026.05 | 82.89 | 0.0006 | |
| GPT-5.3-chatType=Single LLM, In ($/M)=1.75, Out ($/M)=14.002026.05 | 82.29 | 0.0043 | |
| GPT-5.4Type=Single LLM, In ($/M)=2.50, Out ($/M)=15.002026.05 | 81.26 | 0.0064 | |
| Num. TurnsType=Heuristic Router2026.05 | 80.41 | 0.0041 | |
| GPT-5.4-miniType=Single LLM, In ($/M)=0.75, Out ($/M)=4.502026.05 | 80.25 | 0.0021 | |
| GPT-5-nanoType=Single LLM, In ($/M)=0.05, Out ($/M)=0.402026.05 | 79.15 | 0.0002 | |
| LengthType=Heuristic Router2026.05 | 78.75 | 0.0006 | |
| GPT-5.4-nanoType=Single LLM, In ($/M)=0.20, Out ($/M)=1.252026.05 | 78 | 0.0005 | |
| GPT-5-miniType=Single LLM, In ($/M)=0.25, Out ($/M)=2.002026.05 | 77.33 | 0.0009 | |
| Num. ToolsType=Heuristic Router2026.05 | 75.63 | 0.0054 | |
| Qwen-3.5-9BType=Single LLM, In ($/M)=0.05, Out ($/M)=0.152026.05 | 72.4 | 0.0002 | |
| Kimi-K2.5Type=Single LLM, In ($/M)=0.60, Out ($/M)=3.002026.05 | 60.88 | 0.0008 |