Realistic Shopping Tasks on WebShop
35.6Success Rate (%)Cloud-only
Evaluation Results
| Method | Links | |||
|---|---|---|---|---|
| Cloud-onlyCloud model=Qwen-Max2026.05 | 35.6 | 14.7 | 181.5 | |
| HeraRouting method=Dynamic routing2026.05 | 34.1 | 8.1 | 112.2 | |
| FrugalGPTRouting method=Dynamic routing2026.05 | 31.8 | 12.1 | 125.8 | |
| RouteLLMRouting method=Dynamic routing2026.05 | 31.1 | 7.2 | 155.1 | |
| Hybrid LLMRouting method=Dynamic routing2026.05 | 29.1 | 7.7 | 109.3 | |
| Rand. (0.5)Routing probability=0.52026.05 | 27.6 | 8.3 | 108.1 | |
| EagleRouting method=Dynamic routing2026.05 | 26.6 | 8 | 116.6 | |
| Rand. (0.4)Routing probability=0.42026.05 | 24.5 | 6.9 | 97.6 | |
| Rand. (0.3)Routing probability=0.32026.05 | 21.2 | 5.8 | 85.9 | |
| Device-onlyOn-device model=Qwen2.5-7B-Instruct2026.05 | 3.8 | 0 | 61.8 |