Agentic Task Completion on τ2-bench Airline
97Success RateSagaLLM
Evaluation results
| Method | Links | ||||
|---|---|---|---|---|---|
| SagaLLMFramework=SagaLLM2026.05 | 97 | — | 76 | 80,000 | |
| RACFramework=RAC2026.05 | 97 | 81 | 50 | 77,000 | |
| LangGraphFramework=LG2026.05 | 82 | — | 52 | 48,000 | |
| LangGraphFramework=LG(PE), Prompt Engineering=true2026.05 | 82 | — | 69 | 62,000 | |
| IG-clarifierAgent Model=DeepSeek-R1, Clarifier Model=Ours2026.06 | 46 | — | — | — | |
| Qwen3-8BAgent Model=DeepSeek-R1, Clarifier Model=Qwen3-8B2026.06 | 42 | — | — | — | |
| IG-clarifierAgent Model=DeepSeek-V3.2, Clarifier Model=Ours2026.06 | 42 | — | — | — | |
| NoneAgent Model=DeepSeek-R1, Clarifier Model=None2026.06 | 40 | — | — | — | |
| Qwen3-8BAgent Model=DeepSeek-V3.2, Clarifier Model=Qwen3-8B2026.06 | 40 | — | — | — | |
| NoneAgent Model=DeepSeek-V3.2, Clarifier Model=None2026.06 | 30 | — | — | — | |
| IG-clarifierAgent Model=Agent Avg., Clarifier Model=Ours2026.06 | 29.9 | — | — | — | |
| IG-clarifierAgent Model=GLM-4-32B, Clarifier Model=Ours2026.06 | 26 | — | — | — | |
| NoneAgent Model=Agent Avg., Clarifier Model=None2026.06 | 24.5 | — | — | — | |
| NoneAgent Model=GLM-4-32B, Clarifier Model=None2026.06 | 24 | — | — | — | |
| Qwen3-8BAgent Model=Agent Avg., Clarifier Model=Qwen3-8B2026.06 | 24 | — | — | — | |
| Qwen3-8BAgent Model=GLM-4-32B, Clarifier Model=Qwen3-8B2026.06 | 20 | — | — | — | |
| IG-clarifierAgent Model=Qwen3-32B, Clarifier Model=Ours2026.06 | 18 | — | — | — | |
| IG-clarifierAgent Model=Qwen3-8B, Clarifier Model=Ours2026.06 | 17.3 | — | — | — | |
| NoneAgent Model=Qwen3-8B, Clarifier Model=None2026.06 | 14.3 | — | — | — | |
| NoneAgent Model=Qwen3-32B, Clarifier Model=None2026.06 | 14 | — | — | — | |
| Qwen3-8BAgent Model=Qwen3-32B, Clarifier Model=Qwen3-8B2026.06 | 10 | — | — | — | |
| Qwen3-8BAgent Model=Qwen3-8B, Clarifier Model=Qwen3-8B2026.06 | 8 | — | — | — |