User Simulation on τ-USI retail airline tasks (out-of-distribution)
87.4Conv ScoreHuman (upper bound)
Evaluation results
| Method | Links | |||||||
|---|---|---|---|---|---|---|---|---|
| Human (upper bound)Type=Human baseline2026.06 | 87.4 | 97.9 | 88 | 93.5 | 97.4 | 0.081 | 92.7 | |
| OSIM-4B-MidCategory=ODYSSIM (ours), Backbone=4B, Evaluation Mode=zero-shot, Variant=Mid2026.06 | 51.1 | 91.5 | 65.5 | 82.6 | 69 | 0.242 | 72.6 | |
| Qwen3-8B-InstructCategory=Off-the-shelf instruct model, Backbone=8B2026.06 | 50.5 | 74.3 | 80.2 | 80.3 | 72.2 | 0.293 | 71.4 | |
| GPT-5.1Category=Frontier LLM2026.06 | 47.3 | 77.4 | 73.3 | 88.1 | 72.1 | 0.172 | 73.5 | |
| OSIM-8BCategory=ODYSSIM (ours), Backbone=8B, Evaluation Mode=zero-shot2026.06 | 46.4 | 88.2 | 79.7 | 93.2 | 70 | 0.255 | 75.4 | |
| OSIM-8B-MidCategory=ODYSSIM (ours), Backbone=8B, Evaluation Mode=zero-shot, Variant=Mid2026.06 | 45.5 | 83.5 | 63.7 | 76.1 | 69.1 | 0.356 | 67.1 | |
| DeepSeek-V3.1Category=Frontier LLM2026.06 | 45.1 | 86.6 | 74.5 | 87.6 | 74.3 | 0.119 | 76.1 | |
| OSIM-Inst-8BCategory=ODYSSIM (ours), Backbone=8B, Evaluation Mode=zero-shot, Variant=Instruct2026.06 | 45.1 | 85.2 | 74.1 | 83.6 | 74.4 | 0.339 | 71.4 | |
| Gemini-3.1-ProCategory=Frontier LLM2026.06 | 44.1 | 67.1 | 48.9 | 45.3 | 75.1 | 0.101 | 61.7 | |
| CoSER-8BCategory=Specialized behavioral simulator, Backbone=8B2026.06 | 37.8 | 71.5 | 71.6 | 69.9 | 63.3 | 0.109 | 67.2 | |
| OSIM-Inst-4BCategory=ODYSSIM (ours), Backbone=4B, Evaluation Mode=zero-shot, Variant=Instruct2026.06 | 37.8 | 73.4 | 62.6 | 62.1 | 72.1 | 0.236 | 64.1 | |
| Human-Like-7BCategory=Specialized behavioral simulator, Backbone=7B2026.06 | 35.7 | 55 | 51.6 | 65.9 | 72.8 | 0.22 | 59.8 | |
| Claude-Opus-4Category=Frontier LLM2026.06 | 32.6 | 71.9 | 46.6 | 44.9 | 73.4 | 0.139 | 59.2 | |
| UserLM-8BCategory=Specialized behavioral simulator, Backbone=8B2026.06 | 30.8 | 50.8 | 56.8 | 80 | 67.4 | 0.14 | 62 | |
| HumanLM-8BCategory=Specialized behavioral simulator, Backbone=8B2026.06 | 30.1 | 19.5 | 38.5 | 50.7 | 61.6 | 0.192 | 46.9 | |
| OSIM-4BCategory=ODYSSIM (ours), Backbone=4B, Evaluation Mode=zero-shot2026.06 | 30 | 66.3 | 76.2 | 82.2 | 70.5 | 0.242 | 66.8 |