# MHSafeEval: Role-Aware Interaction-Level Evaluation of Mental Health Safety in Large Language Models

Source: [MHSafeEval: Role-Aware Interaction-Level Evaluation of Mental Health Safety in Large Language Models](<https://www.sota2.com/research/paper/69e705027d0172140ec7f19b>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2026-04-22T05:22:24.044Z

Authors: Suhyun Lee, Palakorn Achananuparp, Neemesh Yadav, Ee-Peng Lim, Yang Deng

Published: 2026-04-20T02:37:45.000Z

## Abstract

Large language models (LLMs) are increasingly explored as scalable tools for mental health counseling, yet evaluating their safety remains challenging due to the interactional and context-dependent nature of clinical harm. Existing evaluation frameworks predominantly assess isolated responses using coarse-grained taxonomies or static datasets, limiting their ability to diagnose how harms emerge and accumulate over multi-turn counseling interactions. In this work, we introduce R-MHSafe, a role-aware mental health safety taxonomy that characterizes clinically significant harm in terms of the interactional roles an AI counselor adopts, including perpetrator, instigator, facilitator, or enabler, combined with clinically grounded harm categories. Then, we propose MHSafeEval, a closed-loop, agent-based evaluation framework that formulates safety assessment as trajectory-level discovery of harm through adversarial multi-turn interactions, guided by role-aware modeling. Using R-MHSafe and MHSafeEval, we conduct a large-scale evaluation across state-of-the-art LLMs. Our results reveal substantial role-dependent and cumulative safety failures that are systematically missed by existing static benchmarks, and show that our framework significantly improves failure-mode coverage and diagnostic granularity.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Mental Health Safety Evaluation on MHSafeEval (v1.0)](<https://www.sota2.com/research/sota/mental-health-safety-evaluation-on-mhsafeeval-v1-0.md>): Mental Health Safety Evaluation; MHSafeEval seven mental-health harm categories 1.0; ASR (Gaslighting): not recorded.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2604.17730>)

[PDF](<https://arxiv.org/pdf/2604.17730.pdf>)