Next-action prediction on OPeRA (test)
52.92Action Generation AccAlignUSER+
Evaluation results
| Method | Links | ||||||
|---|---|---|---|---|---|---|---|
| AlignUSER+2026.01 | 52.92 | 71.94 | 66.88 | 80.52 | — | — | |
| AlignUSER2026.01 | 51.47 | 69.81 | 66.29 | 78.07 | — | — | |
| Shop-R1 (Qwen2.5-VL-3B-Instruct)Settings=SFT + RL2025.07 | 38.44 | — | — | — | 57.27 | 57.69 | |
| ContextSim2026.01 | 35.37 | 64.22 | 61.03 | 73.74 | — | — | |
| SimUSER2026.01 | 24.21 | 52.44 | 48.68 | 59.63 | — | — | |
| SimUSER2026.01 | 24.21 | 52.44 | 48.68 | 59.63 | — | — | |
| Agent4Rec2026.01 | 23.09 | 50.05 | 46.37 | 56.7 | — | — | |
| Agent4Rec2026.01 | 23.09 | 50.05 | 46.37 | 56.7 | — | — | |
| RecAgent2026.01 | 22.71 | 49.18 | 45.25 | 54.12 | — | — | |
| RecAgent2026.01 | 22.71 | 49.18 | 45.25 | 54.12 | — | — | |
| GPT-4.1persona=excluded, rationale=included2026.01 | 22.06 | 45.55 | 43.45 | 58.47 | — | — | |
| GPT-4.1Ablation=w/o persona2026.01 | 22.06 | 45.55 | 43.45 | 58.47 | — | — | |
| GPT-4.1persona=included, rationale=included2026.01 | 21.51 | 48.78 | 44.47 | 47.54 | — | — | |
| GPT-4.1Ablation=Full2026.01 | 21.51 | 48.78 | 44.47 | 47.54 | — | — | |
| GPT-4.1persona=included, rationale=excluded2026.01 | 21.28 | 34.93 | 42.63 | 51.17 | — | — | |
| GPT-4.1Ablation=w/o rationale2026.01 | 21.28 | 34.93 | 42.63 | 51.17 | — | — | |
| Qwen2.5-VL-3B-InstructSettings=SFT2025.07 | 20.23 | — | — | — | 60.86 | 53.95 | |
| Claude-3.7-Sonnetpersona=included, rationale=included2026.01 | 10.75 | 31.58 | 27.27 | 43.52 | — | — | |
| Claude-3.7-Sonnetpersona=excluded, rationale=included2026.01 | 10.75 | 25.33 | 22.76 | 43.1 | — | — | |
| Claude-3.7Ablation=Full, Model Detail=Claude-3.7-Sonnet2026.01 | 10.75 | 31.58 | 27.27 | 43.52 | — | — | |
| Claude-3.7Ablation=w/o persona, Model Detail=Claude-3.7-Sonnet2026.01 | 10.75 | 25.33 | 22.76 | 43.1 | — | — | |
| Claude-3.7-Sonnetpersona=included, rationale=excluded2026.01 | 10.08 | 26.06 | 20.29 | 43.1 | — | — | |
| Claude-3.7Ablation=w/o rationale, Model Detail=Claude-3.7-Sonnet2026.01 | 10.08 | 26.06 | 20.29 | 43.1 | — | — | |
| Llama-3.3-70B-Instructpersona=included, rationale=excluded2026.01 | 8.76 | 23.6 | 19.22 | 34.19 | — | — | |
| Llama-3.3Ablation=w/o rationale, Model Detail=Llama-3.3-70B-Instruct2026.01 | 8.76 | 23.6 | 19.22 | 34.19 | — | — | |
| Llama-3.3-70B-Instructpersona=included, rationale=included2026.01 | 8.31 | 24.29 | 19.99 | 36.64 | — | — | |
| Llama-3.3-70B-Instructpersona=excluded, rationale=included2026.01 | 8.31 | 23.69 | 18.59 | 33.21 | — | — | |
| Llama-3.3Ablation=Full, Model Detail=Llama-3.3-70B-Instruct2026.01 | 8.31 | 24.29 | 19.99 | 36.64 | — | — | |
| Llama-3.3Ablation=w/o persona, Model Detail=Llama-3.3-70B-Instruct2026.01 | 8.31 | 23.69 | 18.59 | 33.21 | — | — | |
| Claude-3.5-SonnetSettings=Zero-shot Prompt2025.07 | 7.66 | — | — | — | 58.83 | 45.61 | |
| Qwen2.5-VL-3B-InstructSettings=Zero-shot Prompt2025.07 | 6.41 | — | — | — | 34.45 | 38.79 |