Evaluator Alignment with Human Judgments on DynSess-Eval 8 personas (test)
67Role Consistency Rank AccDynSess-Eval
Evaluation results
| Method | Links | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| DynSess-EvalEval Type=Session-Level2026.05 | 67 | 83 | 26 | 77 | 27 | 33 | 73 | 22 | |
| DynSess-Eval (w/o Rubric-Anchored)Eval Type=Session-Level, Ablation=w/o Rubric-Anchored2026.05 | 63 | 73 | 49 | 67 | 59 | 44 | 70 | 59 | |
| CharacterRMEval Type=Turn-Level2026.05 | 57 | — | — | 50 | 37 | 37 | 50 | 51 | |
| CharacterArenaEval Type=Trajectory Rank2026.05 | 57 | 53 | — | — | — | — | 60 | — | |
| CharacterJudgeEval Type=Turn-Level2026.05 | 53 | 27 | 51 | 40 | 45 | 41 | 57 | 54 | |
| RMTBenchEval Type=Pseudo-Session2026.05 | 50 | 33 | 53 | 57 | 39 | 41 | 53 | 50 | |
| DynSess-Eval (w/o Session-Level eval)Eval Type=Turn-Level, Ablation=w/o Session-Level eval2026.05 | 43 | 20 | 52 | 53 | 67 | 45 | 63 | 67 |