Agentic Tool-use on tau2-bench Airline
63.5Pass@1CODEDELEGATOR
Evaluation results
| Method | Links | |||||
|---|---|---|---|---|---|---|
| CODEDELEGATORModel=DeepseekV3.22026.01 | 63.5 | 53.7 | 48.5 | 46 | — | |
| GPT-5Model Category=Proprietary Models2026.03 | 62.5 | 55.3 | 51 | 48 | — | |
| CODEDELEGATORModel=GPT5.02026.01 | 62 | 50.7 | 44.5 | 42 | — | |
| CodeActModel=DeepseekV3.22026.01 | 59 | 45.3 | 37 | 32 | — | |
| ReActModel=DeepseekV3.22026.01 | 58.5 | 47 | 40.5 | 36 | — | |
| +PACESLM=Qwen3.5-9B2026.05 | 57.5 | 45.7 | 39.5 | 36 | — | |
| ReActModel=GPT5.02026.01 | 57 | 49.3 | 45 | 42 | — | |
| CodeActModel=GPT5.02026.01 | 56.5 | 47.3 | 41.5 | 40 | — | |
| +PESLM=Qwen3.5-9B2026.05 | 55 | 45 | 39 | 34 | — | |
| VanillaSLM=Qwen3.5-9B2026.05 | 53.5 | 41.7 | 35.5 | 32 | — | |
| Qwen3-235B-A22B-Inst.Model Category=Open-Source Models, Parameter Scale=≥70B2026.03 | 48.5 | 37.7 | 31 | 26 | — | |
| Simia-Tau-32BModel Category=Open-Source Models, Parameter Scale=~30B2026.03 | 48.5 | 36.3 | 32 | 30 | — | |
| GPT-4oModel Category=Proprietary Models2026.03 | 47.6 | 35.7 | 30 | 26.5 | — | |
| xLAM-2-32b-fc-rModel Category=Open-Source Models, Parameter Scale=~30B2026.03 | 47 | 36.3 | 31.5 | 28 | — | |
| Sonnet 3.52026.05 | 46 | 32.6 | 26.3 | 22.5 | — | |
| xLAM-2-70b-fc-rModel Category=Open-Source Models, Parameter Scale=≥70B2026.03 | 43.3 | 31.2 | 25.2 | 20.7 | — | |
| Simia-Tau-RL-8BModel Category=Open-Source Models, Parameter Scale=≤8B2026.03 | 43 | 34 | 29 | 26 | — | |
| CoVe-4BModel Category=Open-Source Models, Parameter Scale=≤8B2026.03 | 43 | 30 | 23.5 | 20 | — | |
| Simia-Tau-8BModel Category=Open-Source Models, Parameter Scale=≤8B2026.03 | 42.5 | 33 | 29 | 26 | — | |
| GPT-4o2026.05 | 42 | 27.3 | 22 | 20 | — | |
| Simia-Tau-3BModel Category=Open-Source Models, Parameter Scale=≤8B2026.03 | 36 | — | — | — | — | |
| GEM-32BModel Category=Open-Source Models, Parameter Scale=~30B2026.03 | 35.5 | — | — | — | — | |
| Qwen3-30B-A3B-Inst.Model Category=Open-Source Models, Parameter Scale=~30B2026.03 | 33.5 | 20 | 13.5 | 10 | — | |
| xLAM-2-8b-fc-rModel Category=Open-Source Models, Parameter Scale=≤8B2026.03 | 33.5 | 19.2 | 12.5 | 9 | — | |
| +PACESLM=Qwen3-4B2026.05 | 27.5 | 15.3 | 10 | 6 | — | |
| VanillaSLM=Qwen3-4B2026.05 | 26.5 | 16 | 13 | 12 | — | |
| Qwen3-4B-Inst.Model Category=Open-Source Models, Parameter Scale=≤8B2026.03 | 25 | 17 | 13 | 10 | — | |
| xLAM-2-3b-fc-rModel Category=Open-Source Models, Parameter Scale=≤8B2026.03 | 24 | 12.7 | 7.5 | 4 | — | |
| +PESLM=Qwen3-4B2026.05 | 23.5 | 9.7 | 5 | 4 | — | |
| GEM-8BModel Category=Open-Source Models, Parameter Scale=≤8B2026.03 | 22 | — | — | — | — | |
| DeepSeek-V3.2Mode=Non-Reasoning, Architecture=MoE, Total Params=671B, Activated Params=37B2026.01 | — | — | — | — | 62.6 | |
| EXAONE 4.0Mode=Non-Reasoning, Architecture=Dense, Total Params=32B, Activated Params=32B2026.01 | — | — | — | — | 16 | |
| K-EXAONEMode=Non-Reasoning, Architecture=MoE, Total Params=236B, Activated Params=23B2026.01 | — | — | — | — | 42.6 | |
| Qwen3-235B-A22B Instruct-2507Architecture=MoE, Total Params=235B, Activated Params=22B2026.01 | — | — | — | — | 50 |