Role-Play Evaluation on CoSER
62.43Avg ScoreClaude-4.5-Opus
Evaluation results
| Method | Links | ||||||
|---|---|---|---|---|---|---|---|
| Claude-4.5-Opus2026.01 | 62.43 | 63.74 | 64.28 | 58.45 | 63.24 | — | |
| Claude 4.5-OpusJudge Model=Qwen3-235B2026.06 | 62.43 | 63.74 | 64.28 | 58.45 | 63.24 | — | |
| Gemini-3-Pro2026.01 | 61.8 | 65.95 | 60.42 | 58.34 | 62.49 | — | |
| Gemini 3-ProJudge Model=Qwen3-235B2026.06 | 61.8 | 65.95 | 60.42 | 58.34 | 62.49 | — | |
| GPT-5.12026.01 | 61.1 | 64.95 | 53.99 | 60.13 | 65.35 | — | |
| Gemini-2.5-Pro2026.01 | 60.68 | 61.05 | 60.8 | 57.48 | 63.4 | — | |
| DeepSeek-v3.22026.01 | 58.68 | 55.85 | 57.07 | 57.44 | 64.35 | — | |
| MiniMax-M2-her2026.01 | 57.3 | 60.03 | 50.11 | 49.3 | 69.77 | — | |
| DeepSeek-v3.12026.01 | 53.5 | 50.15 | 53.18 | 53.93 | 56.72 | — | |
| HER-RLLearning Strategy=Reinforcement Learning2026.01 | 53.12 | 54.33 | 47.26 | 52.78 | 58.12 | — | |
| HER-SFTLearning Strategy=Supervised Fine-Tuning2026.01 | 50.92 | 50.52 | 45.99 | 49.78 | 57.37 | — | |
| Qwen3.5 397B-AgentopiaJudge Model=Qwen3-235B2026.06 | 49.16 | 41.02 | 49.67 | 46.93 | 59.01 | — | |
| Qwen2.5-7B-Instruct-CoSERModel Version=Role-Playing Specialized2026.06 | 49.03 | 54.8 | 37.07 | 45.17 | 59.08 | — | |
| Qwen3-4B-Instruct + Psy-CoT + RAPOReasoning Strategy=Psy-CoT, RL Algorithm=RAPO2026.06 | 47.87 | 55.41 | 40.8 | 39.32 | 55.94 | — | |
| Qwen2.5-7B-Instruct + Psy-CoT + RAPOReasoning Strategy=Psy-CoT, RL Algorithm=RAPO2026.06 | 47.74 | 55.77 | 44.67 | 37.5 | 53.02 | — | |
| Llama-3.1-8B-CoSERModel Version=Role-Playing Specialized2026.06 | 47.59 | 49.9 | 37.27 | 42.82 | 60.37 | — | |
| Grok-4.1-FastPerformance Mode=Fast2026.01 | 47.4 | 49.21 | 47.57 | 42.64 | 50.17 | — | |
| Qwen2.5-14B-Instruct-AdaMARPScale=Larger Models2026.06 | 47 | 48.56 | 36.02 | 45.03 | 57.38 | — | |
| Llama-3.1-8B-CrabModel Version=Role-Playing Specialized2026.06 | 46.64 | 52.12 | 37.13 | 40.99 | 56.3 | — | |
| Llama-3.3-70B-InstructScale=Larger Models2026.06 | 46.2 | 49.71 | 36.17 | 40.94 | 57.97 | — | |
| Llama-3.1-70B-CoSERScale=Larger Models2026.06 | 45.89 | 48.17 | 36.26 | 40.95 | 58.2 | — | |
| GPT-5-ChatModel Type=Proprietary & General-Purpose LLMs2026.06 | 45.88 | 46.78 | 35.1 | 42.03 | 59.61 | — | |
| Qwen2.5-7B-Instruct + Psy-CoT + GRPOReasoning Strategy=Psy-CoT, RL Algorithm=GRPO2026.06 | 45.82 | 53.27 | 42.52 | 34.45 | 52.04 | — | |
| DeepSeek-V4-FlashModel Type=Proprietary & General-Purpose LLMs2026.06 | 45.49 | 47.07 | 37.3 | 39.77 | 57.84 | — | |
| Qwen3-4B-Instruct + Psy-CoT + GRPOReasoning Strategy=Psy-CoT, RL Algorithm=GRPO2026.06 | 45.42 | 55.6 | 41 | 34.64 | 50.44 | — | |
| Claude-4.5-Sonnet2026.01 | 45.21 | 47.18 | 36.02 | 47.55 | 50.09 | — | |
| Claude 4.5-SonnetJudge Model=Qwen3-235B2026.06 | 45.21 | 47.18 | 36.02 | 47.55 | 50.09 | — | |
| GPT-4o-miniModel Type=Proprietary & General-Purpose LLMs2026.06 | 45.09 | 47.32 | 40.18 | 40.51 | 52.47 | — | |
| Qwen2.5-7B-Instruct-CrabModel Version=Role-Playing Specialized2026.06 | 44.75 | 55.31 | 32.18 | 40.44 | 51.06 | — | |
| Llama-3.1-8B-Instruct + Psy-CoT + RAPOReasoning Strategy=Psy-CoT, RL Algorithm=RAPO2026.06 | 44.55 | 53.94 | 41.34 | 33.99 | 48.95 | — | |
| Qwen2.5-14B-InstructScale=Larger Models2026.06 | 43.91 | 47.73 | 34.97 | 39.65 | 53.29 | — | |
| Doubao-1.5-Pro-CharacterModel Type=Proprietary & General-Purpose LLMs2026.06 | 43.52 | 48.13 | 32.74 | 40.93 | 52.28 | — | |
| Llama-3.1-8B-Instruct + Psy-CoT + GRPOReasoning Strategy=Psy-CoT, RL Algorithm=GRPO2026.06 | 43.32 | 52.99 | 37.57 | 35.08 | 47.64 | — | |
| Qwen2.5-7B-Instruct + Psy-CoT + DAPOReasoning Strategy=Psy-CoT, RL Algorithm=DAPO2026.06 | 43.1 | 52.09 | 39.08 | 37.19 | 44.03 | — | |
| Qwen2.5-7B-Instruct + Psy-CoT + GSPOReasoning Strategy=Psy-CoT, RL Algorithm=GSPO2026.06 | 42.61 | 51.84 | 37.66 | 32.18 | 48.78 | — | |
| Qwen3.5 397BJudge Model=Qwen3-235B2026.06 | 42.51 | 39.6 | 40.16 | 40.32 | 49.97 | — | |
| Qwen3-4B-InstructReasoning Strategy=Vanilla2026.06 | 42.11 | 47.01 | 35.56 | 35.72 | 50.17 | — | |
| Qwen2.5-7B-Instruct + Psy-CoT + CPOReasoning Strategy=Psy-CoT, RL Algorithm=CPO2026.06 | 41.88 | 52.31 | 36.85 | 28.73 | 49.62 | — | |
| Qwen2.5-7B-InstructReasoning Strategy=Vanilla2026.06 | 41.31 | 48.09 | 33.15 | 37.42 | 46.58 | — | |
| Claude-3.7-ThinkInference Mode=Thought-trace2026.01 | 39.73 | 44.84 | 31 | 42.45 | 40.65 | — | |
| CoSER-70BParameters=70B2026.01 | 35.95 | 35.05 | 31.16 | 32.28 | 45.33 | — | |
| CoSER 70BJudge Model=Qwen3-235B2026.06 | 35.95 | 35.05 | 31.16 | 32.28 | 45.33 | — | |
| GPT-5-MiniModel Class=Mini2026.01 | 32.97 | 38.1 | 24.6 | 27.2 | 42 | — | |
| GPT-5 MiniJudge Model=Qwen3-235B2026.06 | 32.97 | 38.1 | 24.6 | 27.2 | 42 | — | |
| Llama-3.1-8B-InstructReasoning Strategy=Vanilla2026.06 | 31.69 | 37.39 | 25.78 | 30.22 | 33.36 | — | |
| GPT-4o-240806Version=2408062026.01 | 27.69 | 34 | 14.9 | 22.9 | 38.9 | — | |
| GPT-OSS-120BParameters=120B, Type=Open Source2026.01 | 26.12 | 32.8 | 14.8 | 21.5 | 35.4 | — | |
| Qwen3-32BParameters=32B2026.01 | 22.86 | 30.56 | 19.61 | 15.52 | 30.56 | — | |
| Base 8BModel=Qwen3-8B-Base, Parameter Count=8B2026.06 | — | — | — | — | — | 6.1 | |
| Claude Opus 4.72026.06 | — | — | — | — | — | 66.5 | |
| DITTOBackbone=Qwen3-VL-8B-Instruct2026.05 | — | — | — | — | — | 51.2 | |
| Gemini 3.1 Pro2026.06 | — | — | — | — | — | 62.1 | |
| GPT 5.52026.06 | — | — | — | — | — | 66.2 | |
| GPT-5-nano2026.05 | — | — | — | — | — | 35.2 | |
| GPT-5.42026.05 | — | — | — | — | — | 65.9 | |
| GRPOBackbone=Qwen3-VL-8B-Instruct2026.05 | — | — | — | — | — | 54.1 | |
| HER-32B2026.05 | — | — | — | — | — | 41.9 | |
| OSIM 8BModel=OSIM, Parameter Count=8B, Training Stage=Final2026.06 | — | — | — | — | — | 62.6 | |
| OSIM 8B-MidModel=OSIM, Parameter Count=8B, Training Stage=Mid-trained2026.06 | — | — | — | — | — | 24.8 | |
| Others *2026.06 | — | — | — | — | — | 30.3 | |
| Qwen 3.6 Plus2026.06 | — | — | — | — | — | 55.9 | |
| Qwen3 8B InstModel=Qwen3-8B-Instruct, Parameter Count=8B2026.06 | — | — | — | — | — | 43.5 | |
| Qwen3-VL-8B-InstructRole=Base2026.05 | — | — | — | — | — | 43.5 |