Shopping Agent Performance on ShoppingBench 75-problem leak-cluster-guarded held-out
77.3ASRagent_23175
Evaluation results
| Method | Links | ||||||
|---|---|---|---|---|---|---|---|
| agent_23175Source=SN15 leaderboard top miner, Structure=Multi-LLM ensemble2026.06 | 77.3 | — | — | — | — | — | |
| Claude Sonnet 4.6Access=Frontier closed-source2026.06 | 64 | — | — | — | — | — | |
| Qwen3-4B + paper SFT+GRPOModel=Qwen3-4B, Recipe=paper SFT+GRPO2026.06 | 48.7 | — | — | — | — | — | |
| Qwen3-4B + paper SFT (synthetic)Model=Qwen3-4B, Recipe=paper SFT (synthetic)2026.06 | 43.6 | — | — | — | — | — | |
| opd_renderersStack=ORO SFT (final stack)2026.06 | 42.7 | 34.8 | 53.3 | — | — | — | |
| opd_renderers + KTO (β = 0.02, 1 ep)Stack=ORO SFT, beta=0.02, epochs=1 ep2026.06 | 42.7 | — | — | — | — | — | |
| opd_rebalStack=ORO SFT, Method=Voucher-balanced teacher corpus2026.06 | 40 | — | — | — | — | — | |
| GPT-5.5Access=Frontier closed-source2026.06 | 38.7 | — | — | — | — | — | |
| opd_renderers + KTO (β = 0.01, 2 ep)Stack=ORO SFT, beta=0.01, epochs=2 ep2026.06 | 35.1 | — | — | — | — | — | |
| rejection_v2Stack=ORO SFT, Method=Rejection-sampled re-SFT2026.06 | 34.7 | — | — | — | — | — | |
| opd_sonnetStack=ORO SFT, Method=Continued teacher-SFT2026.06 | 34.7 | — | — | — | — | — | |
| Qwen3-4B baseModel=Qwen3-4B, Variant=base2026.06 | 18 | — | — | — | — | — | |
| opd_renderers + Dr. GRPO v19Stack=ORO SFT, Variant=per-step tool-match2026.06 | — | — | — | 0.02 | 14 | — |