User simulator goal alignment on MultiWOZ Challenge
80.1Profile Success Rate (Prof.)Gemma-3-27B-It
Evaluation results
| Method | Links | ||||||
|---|---|---|---|---|---|---|---|
| Gemma-3-27B-ItMethodology=Prompt-Based2025.07 | 80.1 | 50.3 | 95 | 97.7 | 92.1 | 83 | |
| Llama-3.1-8B-ItMethodology=Prompt-Based2025.07 | 73.6 | 35 | 86.1 | 89 | 80.9 | 72.9 | |
| Llama-3.3-70B-ItMethodology=Inference-Time Steering2025.07 | 72.6 | 64.3 | 97 | 97.8 | 91.7 | 84.7 | |
| Gemma-3-27B-ItMethodology=Inference-Time Steering2025.07 | 72.3 | 45.7 | 87 | 89.7 | 74.8 | 73.9 | |
| Qwen-2.5-72B-ItMethodology=Prompt-Based2025.07 | 72 | 31 | 91 | 93.3 | 90.2 | 75.5 | |
| Llama-3.3-70B-ItMethodology=Prompt-Based2025.07 | 72 | 31 | 97.9 | 98.6 | 93.2 | 83.3 | |
| Llama-3.1-8B-ItMethodology=Inference-Time Steering2025.07 | 71.7 | 38 | 85.2 | 88.6 | 74.3 | 71.6 | |
| Qwen-2.5-72B-ItMethodology=Inference-Time Steering2025.07 | 71 | 46.3 | 94.9 | 96.9 | 92.9 | 80.4 | |
| Qwen-2.5-7B-ItMethodology=Cold-Start SFT2025.07 | 60.3 | 38.9 | 88 | 91.1 | 83.1 | 72.3 | |
| Llama-3.1-8B-ItMethodology=GRPO with UGST Rewards2025.07 | 59 | 44 | 99.7 | 99.8 | 97.8 | 80 | |
| Llama-3.1-8B-ItMethodology=Cold-Start SFT2025.07 | 57.1 | 40.1 | 92.1 | 94.9 | 84.3 | 73.7 | |
| Qwen-2.5-7B-ItMethodology=Prompt-Based2025.07 | 54.7 | 18 | 78.4 | 81.9 | 73.5 | 61.3 | |
| Qwen-2.5-7B-ItMethodology=GRPO with UGST Rewards2025.07 | 51.8 | 34 | 98.3 | 98.8 | 93.9 | 75.4 | |
| Qwen-2.5-7B-ItMethodology=Inference-Time Steering2025.07 | 43.4 | 26.3 | 88.1 | 89.8 | 71.1 | 63.7 |