Interactive coding-centric agent tasks on AppWorld
22.6Success RateCloud-only
Evaluation results
| Method | Links | |||
|---|---|---|---|---|
| Cloud-onlyCloud model=Qwen-Max2026.05 | 22.6 | 0.0438 | 332.5 | |
| HeraRouting method=Dynamic routing2026.05 | 19.7 | 0.0313 | 286 | |
| FrugalGPTRouting method=Dynamic routing2026.05 | 18.5 | 0.0375 | 359.5 | |
| RouteLLMRouting method=Dynamic routing2026.05 | 17.6 | 0.0313 | 266.3 | |
| Hybrid LLMRouting method=Dynamic routing2026.05 | 16.2 | 0.0369 | 301.9 | |
| EagleRouting method=Dynamic routing2026.05 | 9.8 | 0.0182 | 317.1 | |
| Rand. (0.4)Routing probability=0.42026.05 | 7.1 | 0.0192 | 363.4 | |
| Rand. (0.5)Routing probability=0.52026.05 | 5.4 | 0.0225 | 392.7 | |
| Rand. (0.3)Routing probability=0.32026.05 | 2.4 | 0.0147 | 304.1 | |
| Device-onlyOn-device model=Qwen2.5-7B-Instruct2026.05 | 0.6 | 0 | 144.2 |