# HealthBench Professional: Evaluating Large Language Models on Real Clinician Chats

Source: [HealthBench Professional: Evaluating Large Language Models on Real Clinician Chats](<https://www.sota2.com/research/paper/69f433b755a3c32af89fd27e>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2026-05-02T05:00:04.163Z

Authors: Rebecca Soskin Hicks, Mikhail Trofimov, Dominick Lim, Rahul K. Arora, Foivos Tsimpourlas, Preston Bowman, Michael Sharman, Chi Tong, Kavin Karthik, Arnav Dugar, Akshay Jagadeesh, Khaled Saab, Johannes Heidecke, Ashley Alexander, Nate Gross, Karan Singhal

Published: 2026-04-30T06:13:01.000Z

## Abstract

Millions of clinicians use ChatGPT to support clinical care, but evaluations of the most common use cases in model-clinician conversations are limited. We introduce HealthBench Professional, an open benchmark for evaluating large language models on real tasks that clinicians bring to ChatGPT in the course of their work. The benchmark is organized around three common use cases central to clinical practice: care consult, writing and documentation, and medical research. Each example includes a physician-authored conversation with ChatGPT for Clinicians and is scored via rubrics written and iteratively adjudicated by three or more physicians across three phases. HealthBench Professional examples were carefully selected for quality, representativeness, and difficulty for OpenAI's current frontier models, to enable continued measurement of progress. Difficult examples for recent OpenAI models were enriched by roughly 3.5 times relative to the candidate pool of 15,079 examples. Additionally, about one-third of examples involve physicians conducting deliberate adversarial testing of models. As a strong baseline, we also collected human physician responses for all tasks (unbounded time, specialist-matched, web access). The best scoring system, GPT-5.4 in ChatGPT for Clinicians, outperforms base GPT-5.4, all other models, and human physicians. We hope HealthBench Professional provides the healthcare AI community a measure to track frontier model progress in real-world clinical tasks and build systems that clinicians can trust to improve care.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Clinical Reasoning on HealthBench Professional (525 cases)](<https://www.sota2.com/research/sota/clinical-reasoning-on-healthbench-professional-525-cases.md>): Clinical Reasoning; HealthBench Professional (525 cases); Overall Score: 59.

- [Medical Question Answering on HealthBench Professional](<https://www.sota2.com/research/sota/medical-question-answering-on-healthbench-professional.md>): Medical Question Answering; HealthBench Professional; Score: 59.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2604.27470>)

[PDF](<https://arxiv.org/pdf/2604.27470.pdf>)