User Simulation Intrinsic Evaluation on PRISM
94.55First-Turn DiversityUserLM-8b
Evaluation results
| Method | Links | ||||||
|---|---|---|---|---|---|---|---|
| UserLM-8bType=Trained user LM, Parameters=8b2025.10 | 94.55 | 2.69 | 63.54 | 80.21 | 93.95 | 94.65 | |
| USP-8bType=Trained user LM, Parameters=8b2025.10 | 94.37 | 6.33 | 21.31 | 77.73 | 98.05 | 97.55 | |
| Human (estimate)Type=Ground Truth2025.10 | 94.01 | 1.68 | — | 90.15 | — | — | |
| UserLM-1bType=Trained user LM, Parameters=1b2025.10 | 90.9 | 3.07 | 56.83 | 78.96 | 91.3 | 93.55 | |
| Llama3.2-1b-InstructType=Prompted assistant LM, Parameters=1b2025.10 | 81.36 | 15.72 | 3.47 | 0.14 | 77.55 | 54.95 | |
| Llama3-8b-InstructType=Prompted assistant LM, Parameters=8b2025.10 | 81.31 | 23.95 | 3.51 | 0.2 | 63.25 | 78.05 | |
| GPT-4oType=Prompted assistant LM2025.10 | 74.42 | 7.68 | 1.38 | 3.31 | 38.85 | 70.95 | |
| GPT-4o-miniType=Prompted assistant LM2025.10 | 66.1 | 9.66 | 15.31 | 0.04 | 80.2 | 88.7 |