Agent Task Completion on τ-bench-retail
70.2Success RateSkillMAS
Evaluation results
| Method | Links | ||||||
|---|---|---|---|---|---|---|---|
| SkillMASEvaluation Source=Internal evaluation2026.05 | 70.2 | — | — | — | — | — | |
| CDMemEvaluation Source=Internal evaluation/rerun2026.05 | 68.4 | — | — | — | — | — | |
| Traj-BootstrapEvaluation Source=Internal evaluation/rerun2026.05 | 68.4 | — | — | — | — | — | |
| ReActEvaluation Source=Internal evaluation/rerun2026.05 | 62.3 | — | — | — | — | — | |
| Direct LLMEvaluation Source=Internal evaluation/rerun2026.05 | 61.4 | — | — | — | — | — | |
| Qwen3-8BAgent Model=DeepSeek-V3.2, Clarifier Model=Qwen3-8B2026.06 | 39.1 | — | — | — | — | — | |
| IG-clarifierAgent Model=DeepSeek-V3.2, Clarifier Model=Ours2026.06 | 37.4 | — | — | — | — | — | |
| NoneAgent Model=DeepSeek-V3.2, Clarifier Model=None2026.06 | 36.5 | — | — | — | — | — | |
| IG-clarifierAgent Model=DeepSeek-R1, Clarifier Model=Ours2026.06 | 33.9 | — | — | — | — | — | |
| NoneAgent Model=DeepSeek-R1, Clarifier Model=None2026.06 | 31.3 | — | — | — | — | — | |
| Qwen3-8BAgent Model=DeepSeek-R1, Clarifier Model=Qwen3-8B2026.06 | 26 | — | — | — | — | — | |
| IG-clarifierAgent Model=Qwen3-32B, Clarifier Model=Ours2026.06 | 24.3 | — | — | — | — | — | |
| IG-clarifierAgent Model=Agent Avg., Clarifier Model=Ours2026.06 | 24.2 | — | — | — | — | — | |
| NoneAgent Model=Qwen3-32B, Clarifier Model=None2026.06 | 23.4 | — | — | — | — | — | |
| NoneAgent Model=Agent Avg., Clarifier Model=None2026.06 | 22.8 | — | — | — | — | — | |
| Qwen3-8BAgent Model=Agent Avg., Clarifier Model=Qwen3-8B2026.06 | 21.9 | — | — | — | — | — | |
| Qwen3-8BAgent Model=Qwen3-32B, Clarifier Model=Qwen3-8B2026.06 | 21.7 | — | — | — | — | — | |
| None (Full User Intent)Clarify Strategy=None (Full User Intent)2026.06 | 20 | — | — | — | — | — | |
| DeepSeek-V3.1Clarify Strategy=DeepSeek-V3.12026.06 | 19.1 | — | — | — | — | — | |
| IG-clarifierAgent Model=Qwen3-8B, Clarifier Model=Ours2026.06 | 18.3 | — | — | — | — | — | |
| Ours (Qwen3-1.7B-trained)Clarify Strategy=Ours (Qwen3-1.7B-trained)2026.06 | 18.3 | — | — | — | — | — | |
| NoneAgent Model=Qwen3-8B, Clarifier Model=None2026.06 | 16.5 | — | — | — | — | — | |
| NoneClarify Strategy=None2026.06 | 16.5 | — | — | — | — | — | |
| Qwen3-14BClarify Strategy=Qwen3-14B2026.06 | 16.5 | — | — | — | — | — | |
| Qwen3-8BAgent Model=Qwen3-8B, Clarifier Model=Qwen3-8B2026.06 | 15.7 | — | — | — | — | — | |
| Qwen3-8BClarify Strategy=Qwen3-8B2026.06 | 15.7 | — | — | — | — | — | |
| Qwen3-30BClarify Strategy=Qwen3-30B2026.06 | 15.7 | — | — | — | — | — | |
| DeepSeek-R1Clarify Strategy=DeepSeek-R12026.06 | 14.8 | — | — | — | — | — | |
| Qwen3-8BAgent Model=GLM-4-32B, Clarifier Model=Qwen3-8B2026.06 | 7 | — | — | — | — | — | |
| IG-clarifierAgent Model=GLM-4-32B, Clarifier Model=Ours2026.06 | 7 | — | — | — | — | — | |
| NoneAgent Model=GLM-4-32B, Clarifier Model=None2026.06 | 6.1 | — | — | — | — | — | |
| FAMABackbone=Qwen3-14B2026.04 | — | 37.9 | 25.7 | 19.7 | 16.3 | 14.7 | |
| ReActBackbone=Qwen3-14B2026.04 | — | 25.2 | 17.8 | 14.7 | 13.2 | 12.1 | |
| SRBackbone=Qwen3-14B2026.04 | — | 32.1 | 19.3 | 14.69 | 12.52 | 11.3 |