Character Roleplay on ARCANE validated slice (5 novels) (Out-of-world)
68APFARCANE
Evaluation results
| Method | Links | ||||
|---|---|---|---|---|---|
| ARCANESize=32B, Mode=Arc2026.06 | 68 | 65.7 | 61.6 | 59.8 | |
| DeepSeek-V4Size=Pro, Mode=Arc2026.06 | 64.1 | 63.7 | 57 | 55.8 | |
| ARCANESize=8B, Mode=Arc2026.06 | 62.5 | 60.5 | 54.5 | 54.6 | |
| DeepSeek-V4Size=Flash, Mode=Arc2026.06 | 62.2 | 62 | 54.5 | 51.4 | |
| DeepSeek-V4Size=Flash, Mode=Summary2026.06 | 57.2 | 57.6 | 49 | 47.7 | |
| DeepSeek-V4Size=Pro, Mode=Summary2026.06 | 57.1 | 57.4 | 48.8 | 46.5 | |
| DeepSeek-V4Size=Flash, Mode=LifeChoice2026.06 | 56.8 | 57 | 48.2 | 46.5 | |
| DeepSeek-V4Size=Pro, Mode=LifeChoice2026.06 | 56.7 | 56.8 | 48.7 | 47.6 | |
| ARCANESize=32B, Mode=Vanilla2026.06 | 56.5 | 56.2 | 48.6 | 41.5 | |
| ARCANESize=32B, Mode=TimeChara2026.06 | 56.4 | 56.1 | 48.7 | 39.6 | |
| DeepSeek-V4Size=Pro, Mode=RAG2026.06 | 55.7 | 56.1 | 47.6 | 47 | |
| DeepSeek-V4Size=Flash, Mode=Vanilla2026.06 | 54.8 | 55.4 | 46.8 | 44.5 | |
| Qwen3Size=32B, Mode=Arc2026.06 | 54.8 | 54.3 | 45.4 | 41.7 | |
| DeepSeek-V4Size=Flash, Mode=TimeChara2026.06 | 54.5 | 55.2 | 46.3 | 43.5 | |
| ARCANESize=32B, Mode=LifeChoice2026.06 | 54.4 | 53.9 | 45.3 | 41.1 | |
| DeepSeek-V4Size=Flash, Mode=RAG2026.06 | 54 | 54.8 | 45.2 | 44.2 | |
| DeepSeek-V4Size=Pro, Mode=Vanilla2026.06 | 53.4 | 54.1 | 45.2 | 44 | |
| DeepSeek-V4Size=Pro, Mode=TimeChara2026.06 | 53 | 53.7 | 44.8 | 42.5 | |
| ARCANESize=32B, Mode=Summary2026.06 | 51.9 | 51.6 | 42.6 | 40.6 | |
| ARCANESize=32B, Mode=RAG2026.06 | 51.6 | 51.8 | 42.9 | 41.6 | |
| ARCANESize=8B, Mode=Summary2026.06 | 50.3 | 49.6 | 41.2 | 38.3 | |
| ARCANESize=8B, Mode=LifeChoice2026.06 | 50.3 | 49.9 | 41.7 | 37.9 | |
| ARCANESize=8B, Mode=RAG2026.06 | 49.6 | 49.4 | 41.2 | 36.3 | |
| Qwen3Size=32B, Mode=Vanilla2026.06 | 48.6 | 48.6 | 39.2 | 35 | |
| Qwen3Size=32B, Mode=Summary2026.06 | 48.5 | 48.7 | 38.9 | 36.6 | |
| Qwen3Size=32B, Mode=LifeChoice2026.06 | 48.4 | 48.6 | 38.8 | 36.3 | |
| Qwen3Size=32B, Mode=TimeChara2026.06 | 48.4 | 48.9 | 39.3 | 34.4 | |
| Qwen3Size=32B, Mode=RAG2026.06 | 48.3 | 48.7 | 39 | 36.1 | |
| ARCANESize=8B, Mode=TimeChara2026.06 | 47.7 | 47.4 | 39.3 | 36.5 | |
| Qwen3Size=8B, Mode=Arc2026.06 | 47.4 | 47.5 | 37.4 | 36 | |
| ARCANESize=8B, Mode=Vanilla2026.06 | 47.4 | 46.8 | 38.5 | 36.3 | |
| Qwen3Size=8B, Mode=LifeChoice2026.06 | 42.3 | 42.2 | 32.1 | 30 | |
| Qwen3Size=8B, Mode=RAG2026.06 | 42.2 | 42.7 | 32.8 | 30.3 | |
| Qwen3Size=8B, Mode=TimeChara2026.06 | 41.8 | 42.8 | 32.7 | 29.4 | |
| Qwen3Size=8B, Mode=Summary2026.06 | 41.7 | 41.9 | 31.7 | 30.3 | |
| Qwen3Size=8B, Mode=Vanilla2026.06 | 41.4 | 42.4 | 32.1 | 30.6 |