Character Probing on ARCANE In-Scenario The Underdogs, East Lynne (low-popularity unvalidated) benchmark leaderboard · SOTA2 Research

Character Probing on ARCANE In-Scenario The Underdogs, East Lynne (low-popularity unvalidated)

64.7APFDeepSeek-V4 Pro Arc

Evaluation Results

Method	Links
DeepSeek-V4 Pro Arc2026.06		64.7	63.6	57.1	60.5
DeepSeek-V4 Pro LifeChoice2026.06		64.4	63.8	57.8	58.7
DeepSeek-V4 Pro RAG2026.06		64.1	63.1	57.6	58.4
DeepSeek-V4 Flash Arc2026.06		63.4	63	55.3	60.9
DeepSeek-V4 Flash RAG2026.06		62.7	61.9	55.6	59.7
ARCANE 32B RAG2026.06		61.7	60	53.9	53.8
DeepSeek-V4 Flash LifeChoice2026.06		61	60.5	53.3	52.8
DeepSeek-V4 Flash Summary2026.06		60.6	60.2	52.8	56.9
DeepSeek-V4 Pro Summary2026.06		60	60.2	52.3	55.7
ARCANE 32B Arc2026.06		60	58.6	52.1	56.5
ARCANE 8B RAG2026.06		59.1	57.6	51.9	51.2
ARCANE 8B Arc2026.06		58.5	57.2	—	51.6
ARCANE 32B LifeChoice2026.06		57.8	55.9	49.8	53.2
DeepSeek-V4 Flash Vanilla2026.06		56.9	57.7	47.9	51.3
DeepSeek-V4 Flash TimeChara2026.06		56.6	57.3	48.6	51.8
ARCANE 32B Summary2026.06		55.3	54.3	46.3	52.8
Qwen3 32B RAG2026.06		55.2	53.2	45.4	41.3
Qwen3 32B Arc2026.06		55.1	53.5	44	50.7
DeepSeek-V4 Pro TimeChara2026.06		54.9	55.6	46.8	50.4
Qwen3 32B LifeChoice2026.06		54.8	54.1	46	45.9
DeepSeek-V4 Pro Vanilla2026.06		54.3	55.2	45.9	51.1
ARCANE 8B LifeChoice2026.06		53.7	52.6	45	49
ARCANE 8B Summary2026.06		52.8	51.4	44.5	46.1
Qwen3 32B Summary2026.06		50.5	50	40.3	42.7
Qwen3 8B RAG2026.06		49.2	48	39.8	37.7
ARCANE 32B Vanilla2026.06		48.9	49.3	39.9	41.6
ARCANE 32B TimeChara2026.06		47.4	47.9	38.6	41.6
Qwen3 8B LifeChoice2026.06		47.2	45.6	37	42
Qwen3 8B Arc2026.06		46.7	46.6	36.1	37.5
ARCANE 8B TimeChara2026.06		44.4	45.1	36	38.6
ARCANE 8B Vanilla2026.06		44.2	43.6	34.9	38.3
Qwen3 8B Summary2026.06		43.3	42.3	33	33.5
Qwen3 32B TimeChara2026.06		42.6	42.9	32.7	34.8
Qwen3 32B Vanilla2026.06		40.1	40.4	30	33.2
Qwen3 8B TimeChara2026.06		38.7	40.3	29.5	31
Qwen3 8B Vanilla2026.06		35.5	36.7	26.1	27.8