LLM Agent Evaluation on tau-bench Airline
78Pass@4NoisyAgent
Evaluation results
| Method | Links | |||||||||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| NoisyAgentbackbone=Qwen3-32B2026.05 | 78 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 49.5 | |
| DAPObackbone=Qwen3-32B2026.05 | 76 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 47.5 | |
| GSPObackbone=Qwen3-32B2026.05 | 74 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 46.5 | |
| GRPObackbone=Qwen3-32B2026.05 | 72 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 45 | |
| NoisyAgentbackbone=Qwen3-8B2026.05 | 68 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 40 | |
| DAPObackbone=Qwen3-8B2026.05 | 66 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 38 | |
| Qwen3-32Bbackbone=Qwen3-32B2026.05 | 66 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 38 | |
| GSPObackbone=Qwen3-8B2026.05 | 64 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 37.5 | |
| GRPObackbone=Qwen3-8B2026.05 | 62 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 36.5 | |
| Qwen3-8Bbackbone=Qwen3-8B2026.05 | 52 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 27 | |
| GPT-5k=42026.03 | 44 | 60 | 23.1 | 146.7 | 8.7 | 67 | 94 | 5.27 | 131 | 97 | 96 | 84 | 88 | 93 | 99 | 30 | 1 | 1 | 1 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Kimi-K2k=42026.03 | 28 | 48 | 23.2 | 50.2 | 7 | 73.1 | 76 | 4.58 | 63 | 91 | 92 | 77 | 81 | 96 | 91 | 70 | 1 | 1 | 1 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| FAMABackbone Model=Qwen3-4B-Instruct2026.04 | 26.7 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 37.6 | 32 | 28.3 | 26 | — | |
| ReActBackbone Model=Qwen3-4B-Instruct2026.04 | 26.4 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 32 | 28 | 26.8 | 26 | — | |
| IRMABackbone Model=Qwen3-14B2026.04 | 19.6 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 36.4 | 26.4 | 22 | 18 | — | |
| Mistralk=42026.03 | 18 | 40 | 23.2 | 25.6 | 7.6 | 69.3 | 89 | 5.76 | 91 | 74 | 99 | 36 | 49 | 79 | 45 | 28 | 1 | 1 | 1 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| FAMABackbone Model=Qwen3-32B2026.04 | 18 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 26.8 | 20 | 18.4 | 18 | — | |
| FAMABackbone Model=Qwen2.5-72B-Instruct2026.04 | 18 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 29.2 | 21.2 | 18.8 | 18 | — | |
| FAMABackbone Model=Qwen3-14B2026.04 | 16.6 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 36.8 | 23.79 | 18.2 | 16 | — | |
| FCBackbone Model=Qwen3-4B-Instruct2026.04 | 15.2 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 27.6 | 19.9 | 17 | 14 | — | |
| ReActBackbone Model=Qwen3-32B2026.04 | 14.8 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 30.4 | 20 | 16.2 | 14 | — | |
| IRMABackbone Model=Qwen3-4B-Instruct2026.04 | 13.2 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 30 | 19.6 | 15.4 | 12 | — | |
| ReActBackbone Model=Qwen2.5-72B-Instruct2026.04 | 12.8 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 24.4 | 18.79 | 15.6 | 10 | — | |
| IRMABackbone Model=Qwen2.5-72B-Instruct2026.04 | 11.6 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 26.4 | 16.7 | 13.6 | 10 | — | |
| FCBackbone Model=Qwen3-32B2026.04 | 10.8 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 17.6 | 13.6 | 12 | 10 | — | |
| IRMABackbone Model=Qwen3-32B2026.04 | 10 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 24.4 | 15.9 | 12.4 | 8 | — | |
| ReActBackbone Model=Qwen3-14B2026.04 | 8.4 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 18.4 | 12 | 9.4 | 8 | — | |
| FCBackbone Model=Qwen3-14B2026.04 | 8 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 14 | 9.6 | 8.4 | 8 | — | |
| FCBackbone Model=Qwen2.5-72B-Instruct2026.04 | 2.4 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 15.2 | 7.6 | 4 | 2 | — | |
| ARESStrategy=Adaptive Selection, Backbone LLM=gpt-oss-20b2026.03 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 36 | — | 12.3 | 678 | — | 13,577 | — | 1,100 | — | — | — | — | — | — | |
| Prompting-Based (Gemini 3 Pro)Strategy=Prompting-Based, Model=Gemini 3 Pro, Backbone LLM=gpt-oss-20b2026.03 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 36 | 0 | 12.6 | 239 | 439 | 4,779 | 8,798 | 379 | 721 | — | — | — | — | — | |
| Prompting-Based (GPT 5)Strategy=Prompting-Based, Model=GPT 5, Backbone LLM=gpt-oss-20b2026.03 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 34 | 2 | 12.4 | 396 | 282 | 7,927 | 5,650 | 641 | 459 | — | — | — | — | — | |
| Rule-Based (High)Strategy=Rule-Based, Effort Level=High, Backbone LLM=gpt-oss-20b2026.03 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 38 | 2 | 10.6 | 873 | 195 | 17,472 | 3,895 | 1,654 | 554 | — | — | — | — | — | |
| Rule-Based (Low)Strategy=Rule-Based, Effort Level=Low, Backbone LLM=gpt-oss-20b2026.03 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 32 | 4 | 10.8 | 12 | 666 | 250 | 13,327 | 23 | 1,077 | — | — | — | — | — | |
| Rule-Based (Medium)Strategy=Rule-Based, Effort Level=Medium, Backbone LLM=gpt-oss-20b2026.03 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 42 | 6 | 13.2 | 98 | 580 | 1,961 | 11,616 | 148 | 952 | — | — | — | — | — | |
| Rule-Based (Random)Strategy=Rule-Based, Effort Level=Random, Backbone LLM=gpt-oss-20b2026.03 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 34 | 2 | 13.5 | 521 | 157 | 10,437 | 3,140 | 770 | 330 | — | — | — | — | — |