User simulator goal alignment on τ-Bench Retail
94.5User Profile AdherenceGemma-3-27B-It
Evaluation results
| Method | Links | ||||||
|---|---|---|---|---|---|---|---|
| Gemma-3-27B-ItEvaluation Paradigm=Inference-Time Steering2025.07 | 94.5 | 56.6 | 97.1 | 99.5 | 89.8 | 87.5 | |
| Llama-3.3-70B-ItEvaluation Paradigm=Inference-Time Steering2025.07 | 92.7 | 59.2 | 96 | 98.4 | 100 | 89.3 | |
| Llama-3.3-70B-ItEvaluation Paradigm=Prompt-Based2025.07 | 91.7 | 46.1 | 99.6 | 100 | 100 | 87.5 | |
| Gemma-3-27B-ItEvaluation Paradigm=Prompt-Based2025.07 | 91 | 39.5 | 97.8 | 99.5 | 96.3 | 84.8 | |
| Qwen-2.5-72B-ItEvaluation Paradigm=Prompt-Based2025.07 | 88.6 | 40.8 | 98.2 | 97.9 | 91.7 | 83.4 | |
| Qwen-2.5-72B-ItEvaluation Paradigm=Inference-Time Steering2025.07 | 85.8 | 48.7 | 97.1 | 98.4 | 98.1 | 85.6 | |
| Qwen-2.5-7B-ItEvaluation Paradigm=Cold-Start SFT2025.07 | 85.8 | 43.4 | 92.4 | 94.7 | 82.4 | 79.7 | |
| Qwen-2.5-7B-ItEvaluation Paradigm=GRPO with UGST Rewards2025.07 | 85.5 | 38.2 | 97.1 | 97.9 | 97.2 | 83.2 | |
| Llama-3.1-8B-ItEvaluation Paradigm=Prompt-Based2025.07 | 84.8 | 36.8 | 97.1 | 98.9 | 96.3 | 82.8 | |
| Llama-3.1-8B-ItEvaluation Paradigm=Inference-Time Steering2025.07 | 84.8 | 52.6 | 95.3 | 98.9 | 97.2 | 85.8 | |
| Llama-3.1-8B-ItEvaluation Paradigm=Cold-Start SFT2025.07 | 84.8 | 47.3 | 95.5 | 97.8 | 94.1 | 83.9 | |
| Llama-3.1-8B-ItEvaluation Paradigm=GRPO with UGST Rewards2025.07 | 84.8 | 47.4 | 98.9 | 99.5 | 98.1 | 85.7 | |
| Qwen-2.5-7B-ItEvaluation Paradigm=Prompt-Based2025.07 | 82 | 40.8 | 96 | 99.5 | 91.7 | 82 | |
| Qwen-2.5-7B-ItEvaluation Paradigm=Inference-Time Steering2025.07 | 73 | 47.4 | 94.9 | 97.9 | 93.5 | 81.4 |