Human agreement evaluation on PsyCrisis (sampled)
0.4837Pearson R (Overall)PsyCrisis LLM-as-Judge (Ours)
Evaluation results
| Method | Links | ||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| PsyCrisis LLM-as-Judge (Ours)Evaluator Model=gpt-4o-2024-08-062025.08 | 0.4837 | 0.4398 | 0.4769 | 0.563 | 0.4534 | 0.4348 | 0.4565 | 0.5135 | 0.4063 | 0.3917 | 0.4225 | 0.4599 | 79.1 | 78.76 | 84.39 | 81.93 | |
| GeneralEvaluator Model=gpt-4o-2024-08-06, Prompt Type=Holistic prompt without structured criteria2025.08 | 0.27 | 0.1519 | 0.1257 | 0.3856 | 0.2498 | 0.1544 | 0.1211 | 0.3534 | 0.2344 | 0.1444 | 0.1162 | 0.3307 | 84.43 | 79.06 | 77.1 | 86.95 | |
| RulesEvaluator Model=gpt-4o-2024-08-06, Prompt Type=Binary scores across five predefined dimensions2025.08 | 0.1524 | 0.1732 | -0.061 | 0.1855 | 0.1548 | 0.1783 | -0.039 | 0.1758 | 0.1342 | 0.1544 | -0.037 | 0.1541 | 59.83 | 61.09 | 46.97 | 61.45 |