Conversational Agent Evaluation on tau2 Airline (Success Rate)
78Success RatePass@N (oracle)
Evaluation results
| Method | Links | |
|---|---|---|
| Pass@N (oracle)Training=false, Backbone=Qwen3.5-9B2026.06 | 78 | |
| DeepConf B10Training=false, Backbone=Qwen3.5-9B2026.06 | 72 | |
| Progress AdvantageTraining=false, Backbone=Qwen3.5-9B2026.06 | 72 | |
| Mean-of-NTraining=false, Backbone=Qwen3.5-9B2026.06 | 64.8 | |
| WildReward-8BTraining=true, Backbone=Qwen3.5-9B2026.06 | 64 | |
| ThinkPRM-7BTraining=true, Backbone=Qwen3.5-9B2026.06 | 64 | |
| Self-CertaintyTraining=false, Backbone=Qwen3.5-9B2026.06 | 64 | |
| DeepConf TailTraining=false, Backbone=Qwen3.5-9B2026.06 | 62 | |
| Greedy DecodingTraining=false, Backbone=Qwen3.5-9B2026.06 | 60 | |
| Pass@N (oracle)Training=false, Backbone=Gemma4-4B2026.06 | 58 | |
| ThinkPRM-14BTraining=true, Backbone=Qwen3.5-9B2026.06 | 58 | |
| Progress AdvantageTraining=false, Backbone=Gemma4-4B2026.06 | 48 | |
| DeepConf TailTraining=false, Backbone=Gemma4-4B2026.06 | 36 | |
| Mean-of-NTraining=false, Backbone=Gemma4-4B2026.06 | 34.5 | |
| Greedy DecodingTraining=false, Backbone=Gemma4-4B2026.06 | 34 | |
| Self-CertaintyTraining=false, Backbone=Gemma4-4B2026.06 | 34 | |
| ThinkPRM-7BTraining=true, Backbone=Gemma4-4B2026.06 | 30 | |
| WildReward-8BTraining=true, Backbone=Gemma4-4B2026.06 | 28 | |
| ThinkPRM-14BTraining=true, Backbone=Gemma4-4B2026.06 | 28 | |
| DeepConf B10Training=false, Backbone=Gemma4-4B2026.06 | 28 |