Agentic Tool-use on τ2-Bench (Retail and Telecom)
85.79Overall Success RateClaude-Opus-4.5
Evaluation results
| Method | Links | |||
|---|---|---|---|---|
| Claude-Opus-4.5Temperature=0, User Simulator=GPT-5.1, Trials=42026.01 | 85.79 | 80.88 | 90.7 | |
| Gemini-3-ProTemperature=0, User Simulator=GPT-5.1, Trials=42026.01 | 83.69 | 77.72 | 89.65 | |
| Claude-Sonnet-4.5Temperature=0, User Simulator=GPT-5.1, Trials=42026.01 | 76.58 | 77.19 | 75.96 | |
| GLM-4.6Temperature=0, User Simulator=GPT-5.1, Trials=42026.01 | 69.63 | 78.95 | 60.31 | |
| Kimi-K2-InstructTemperature=0, User Simulator=GPT-5.1, Trials=42026.01 | 69.52 | 68.64 | 70.39 | |
| Astra-32B-thinking-v1Temperature=0, User Simulator=GPT-5.1, Trials=42026.01 | 63.7 | 75.2 | 52.19 | |
| LoopTool-32BTemperature=0, User Simulator=GPT-5.1, Trials=42026.01 | 60.31 | 72.15 | 48.46 | |
| DeepSeek-V3.2Mode=Non-Reasoning, Architecture=MoE, Total Params=671B, Activated Params=37B2026.01 | 58.3 | — | — | |
| Astra-14B-thinking-v1Temperature=0, User Simulator=GPT-5.1, Trials=42026.01 | 57.69 | 68 | 47.37 | |
| Gemini-2.5-ProTemperature=0, User Simulator=GPT-5.1, Trials=42026.01 | 54.58 | 71.26 | 37.89 | |
| GPT-4.1Temperature=0, User Simulator=GPT-5.1, Trials=42026.01 | 54 | 74 | 34 | |
| Claude-Haiku-4.5Temperature=0, User Simulator=GPT-5.1, Trials=42026.01 | 53.16 | 69.12 | 37.19 | |
| Qwen3-32BTemperature=0, User Simulator=GPT-5.1, Trials=42026.01 | 49.7 | 64.7 | 34.7 | |
| Qwen3-14BTemperature=0, User Simulator=GPT-5.1, Trials=42026.01 | 46.05 | 55 | 37.1 | |
| K-EXAONEMode=Non-Reasoning, Architecture=MoE, Total Params=236B, Activated Params=23B2026.01 | 44 | — | — | |
| Qwen3-235B-A22B Instruct-2507Architecture=MoE, Total Params=235B, Activated Params=22B2026.01 | 32.5 | — | — | |
| EXAONE 4.0Mode=Non-Reasoning, Architecture=Dense, Total Params=32B, Activated Params=32B2026.01 | 16.7 | — | — |