User simulator goal alignment on τ-Bench Airline
98.7User Profile Alignment (Prof.)Gemma-3-27B-It
Evaluation results
| Method | Links | ||||||
|---|---|---|---|---|---|---|---|
| Gemma-3-27B-ItEvaluation Paradigm=Prompt-Based2025.07 | 98.7 | 59 | 97.1 | 97 | 78.6 | 86.1 | |
| Qwen-2.5-7B-ItEvaluation Paradigm=Cold-Start SFT2025.07 | 98.7 | 55.7 | 99 | 100 | 95.2 | 89.7 | |
| Llama-3.3-70B-ItEvaluation Paradigm=Prompt-Based2025.07 | 97.4 | 65.6 | 98.1 | 99 | 92.9 | 90.6 | |
| Llama-3.1-8B-ItEvaluation Paradigm=GRPO with UGST Rewards2025.07 | 96.1 | 62.3 | 100 | 100 | 97.6 | 91.2 | |
| Gemma-3-27B-ItEvaluation Paradigm=Inference-Time Steering2025.07 | 94.8 | 62.3 | 92.4 | 97 | 85.7 | 86.4 | |
| Llama-3.3-70B-ItEvaluation Paradigm=Inference-Time Steering2025.07 | 93.5 | 63.9 | 98.1 | 100 | 97.6 | 90.6 | |
| Qwen-2.5-7B-ItEvaluation Paradigm=GRPO with UGST Rewards2025.07 | 93.5 | 63.9 | 100 | 100 | 100 | 91.5 | |
| Qwen-2.5-72B-ItEvaluation Paradigm=Inference-Time Steering2025.07 | 93.3 | 54.4 | 98 | 97.9 | 97.6 | 88.3 | |
| Llama-3.1-8B-ItEvaluation Paradigm=Inference-Time Steering2025.07 | 92.2 | 57.4 | 93.3 | 98 | 95.2 | 87.2 | |
| Llama-3.1-8B-ItEvaluation Paradigm=Prompt-Based2025.07 | 90.9 | 41 | 97.1 | 99 | 81 | 81.8 | |
| Llama-3.1-8B-ItEvaluation Paradigm=Cold-Start SFT2025.07 | 89.6 | 55.7 | 97.1 | 97 | 97.6 | 87.4 | |
| Qwen-2.5-72B-ItEvaluation Paradigm=Prompt-Based2025.07 | 89.3 | 59.6 | 94.1 | 96.9 | 78.6 | 83.7 | |
| Qwen-2.5-7B-ItEvaluation Paradigm=Prompt-Based2025.07 | 88.3 | 49.2 | 94.3 | 96 | 85.7 | 82.7 | |
| Qwen-2.5-7B-ItEvaluation Paradigm=Inference-Time Steering2025.07 | 77.9 | 55.7 | 96.2 | 98 | 92.9 | 84.1 |