LLM Agent Evaluation on tau-bench Retail
64.8Pass@1EVOTOOL
Evaluation results
| Method | Links | ||||||||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| EVOTOOLBackbone=GPT-4o2026.03 | 64.8 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| AdaPlannerBackbone=GPT-4o2026.03 | 62.2 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| EvoPromptBackbone=GPT-4o2026.03 | 60.8 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| ANYTOOLBackbone=GPT-4o2026.03 | 60.4 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| ReActBackbone=GPT-4o2026.03 | 59.8 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Plan-and-SolveBackbone=GPT-4o2026.03 | 58.4 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| OPROBackbone=GPT-4o2026.03 | 58.2 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| PromptBreederBackbone=GPT-4o2026.03 | 56.4 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| EASYTOOLBackbone=GPT-4o2026.03 | 51.2 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| DRAFTBackbone=GPT-4o2026.03 | 48.4 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| FAMABackbone Model=Qwen2.5-72B-Instruct2026.04 | 44.17 | — | 28.17 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 34.26 | 30.26 | 26.95 | |
| ReActBackbone Model=Qwen2.5-72B-Instruct2026.04 | 43.47 | — | 23.13 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 32.26 | 26.69 | 20.86 | |
| FAMABackbone Model=Qwen3-32B2026.04 | 40.5 | — | 15.3 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 26.9 | 19.9 | 12.2 | |
| IRMABackbone Model=Qwen2.5-72B-Instruct2026.04 | 38.78 | — | 20.34 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 27.73 | 22.95 | 19.13 | |
| ReActBackbone Model=Qwen3-32B2026.04 | 38 | — | 15.3 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 25 | 18.8 | 10 | |
| FAMABackbone Model=Qwen3-14B2026.04 | 37.9 | — | 16.3 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 25.7 | 19.7 | 14.7 | |
| EVOTOOLBackbone=Qwen3-8B2026.03 | 35.9 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| FCBackbone Model=Qwen3-32B2026.04 | 35 | — | 13.1 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 26.6 | 15 | 11 | |
| AdaPlannerBackbone=Qwen3-8B2026.03 | 34.8 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| FAMABackbone Model=Qwen3-4B-Instruct2026.04 | 34.6 | — | 16.3 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 24.1 | 19.3 | 13.9 | |
| CoTBackbone=GPT-4o2026.03 | 34.2 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| EvoPromptBackbone=Qwen3-8B2026.03 | 33.8 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| ReActBackbone=Qwen3-8B2026.03 | 33.1 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| FCBackbone Model=Qwen3-14B2026.04 | 32.6 | — | 15 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 22 | 18 | 13 | |
| Plan-and-SolveBackbone=Qwen3-8B2026.03 | 32 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| OPROBackbone=Qwen3-8B2026.03 | 31.9 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| PromptBreederBackbone=Qwen3-8B2026.03 | 30.6 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| IRMABackbone Model=Qwen3-4B-Instruct2026.04 | 28.86 | — | 10.95 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 17.47 | 13.13 | 9.56 | |
| IRMABackbone Model=Qwen3-14B2026.04 | 28.5 | — | 8.3 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 15.6 | 10.8 | 6.9 | |
| IRMABackbone Model=Qwen3-32B2026.04 | 26.9 | — | 8.5 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 15.2 | 10.9 | 6.9 | |
| ReActBackbone Model=Qwen3-14B2026.04 | 25.2 | — | 13.2 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 17.8 | 14.7 | 12.1 | |
| FCBackbone Model=Qwen3-4B-Instruct2026.04 | 24.9 | — | 10.6 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 16.7 | 13 | 9 | |
| ANYTOOLBackbone=Qwen3-8B2026.03 | 23.5 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| EASYTOOLBackbone=Qwen3-8B2026.03 | 20.9 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| FCBackbone Model=Qwen2.5-72B-Instruct2026.04 | 19.65 | — | 5.21 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 10.34 | 6.78 | 4.34 | |
| DRAFTBackbone=Qwen3-8B2026.03 | 17.8 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| ReActBackbone Model=Qwen3-4B-Instruct2026.04 | 17.22 | — | 9.57 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 12.35 | 10.61 | 8.7 | |
| CoTBackbone=Qwen3-8B2026.03 | 14.6 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| ARESStrategy=Adaptive Selection, Backbone LLM=gpt-oss-20b2026.03 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 54.8 | — | 14.6 | 652 | — | 5,677 | — | 388 | — | — | — | — | |
| GPT-5k=42026.03 | — | 79 | 58 | 23.9 | 123.6 | 7.8 | 71.2 | 98 | 5.36 | 119 | 98 | 98 | 91 | 83 | 99 | 97 | 23 | 1 | 1 | 1 | — | — | — | — | — | — | — | — | — | — | — | — | |
| Kimi-K2k=42026.03 | — | 61 | 31 | 23.8 | 29.4 | 6.6 | 69.6 | 89 | 5.23 | 67 | 91 | 94 | 73 | 80 | 97 | 91 | 67 | 1 | 1 | 1 | — | — | — | — | — | — | — | — | — | — | — | — | |
| Mistralk=42026.03 | — | 68 | 46 | 27.6 | 29.3 | 7.9 | 89.2 | 86 | 7.05 | 80 | 85 | 98 | 50 | 66 | 93 | 59 | 23 | 1 | 1 | 1 | — | — | — | — | — | — | — | — | — | — | — | — | |
| Prompting-Based (Gemini 3 Pro)Strategy=Prompting-Based, Model=Gemini 3 Pro, Backbone LLM=gpt-oss-20b2026.03 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 46.1 | 8.7 | 14.7 | 128 | 524 | 1,115 | 4,562 | 76 | 312 | — | — | — | |
| Prompting-Based (GPT 5)Strategy=Prompting-Based, Model=GPT 5, Backbone LLM=gpt-oss-20b2026.03 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 40 | 14.8 | 14 | 132 | 520 | 1,148 | 4,529 | 81 | 307 | — | — | — | |
| Rule-Based (High)Strategy=Rule-Based, Effort Level=High, Backbone LLM=gpt-oss-20b2026.03 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 54.8 | 0 | 13.8 | 1,007 | 355 | 8,756 | 3,079 | 634 | 246 | — | — | — | |
| Rule-Based (Low)Strategy=Rule-Based, Effort Level=Low, Backbone LLM=gpt-oss-20b2026.03 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 35 | 19.8 | 13.5 | 25 | 627 | 223 | 5,454 | 15 | 373 | — | — | — | |
| Rule-Based (Medium)Strategy=Rule-Based, Effort Level=Medium, Backbone LLM=gpt-oss-20b2026.03 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 47.3 | 7.5 | 14.5 | 137 | 515 | 1,198 | 4,479 | 82 | 306 | — | — | — | |
| Rule-Based (Random)Strategy=Rule-Based, Effort Level=Random, Backbone LLM=gpt-oss-20b2026.03 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 43.5 | 11.3 | 14.5 | 359 | 293 | 3,126 | 2,550 | 215 | 172 | — | — | — |