EHRSHOT: An EHR Benchmark for Few-Shot Evaluation of Foundation Models
About
While the general machine learning (ML) community has benefited from public datasets, tasks, and models, the progress of ML in healthcare has been hampered by a lack of such shared assets. The success of foundation models creates new challenges for healthcare ML by requiring access to shared pretrained models to validate performance benefits. We help address these challenges through three contributions. First, we publish a new dataset, EHRSHOT, which contains deidentified structured data from the electronic health records (EHRs) of 6,739 patients from Stanford Medicine. Unlike MIMIC-III/IV and other popular EHR datasets, EHRSHOT is longitudinal and not restricted to ICU/ED patients. Second, we publish the weights of CLMBR-T-base, a 141M parameter clinical foundation model pretrained on the structured EHR data of 2.57M patients. We are one of the first to fully release such a model for coded EHR data; in contrast, most prior models released for clinical data (e.g. GatorTron, ClinicalBERT) only work with unstructured text and cannot process the rich, structured data within an EHR. We provide an end-to-end pipeline for the community to validate and build upon its performance. Third, we define 15 few-shot clinical prediction tasks, enabling evaluation of foundation models on benefits such as sample efficiency and task adaptation. Our model and dataset are available via a research data use agreement from our website: https://ehrshot.stanford.edu. Code to reproduce our results are available at our Github repo: https://github.com/som-shahlab/ehrshot-benchmark
Key insights
What problem does EHRSHOT address?
EHRSHOT tackles the lack of shared EHR datasets and pretrained foundation model weights for healthcare ML. It releases a longitudinal structured EHR dataset of 6,739 patients, a 141M parameter pretrained model, and 15 few-shot clinical prediction tasks.
How does it differ from prior EHR benchmarks?
Unlike ICU-only datasets like MIMIC-IV, EHRSHOT includes longitudinal data across all departments. It uniquely combines open dataset, preprocessing code, and released model weights, unlike prior benchmarks that rarely publish pretrained FM weights.
What are the practical uses of these findings?
EHRSHOT provides a reproducible benchmark for evaluating clinical foundation models in low-label settings. The released CLMBR-T-base model demonstrates AUROC and AUPRC gains over a count-based GBM in few-shot settings, useful for data-scarce clinical prediction research.
What limitations do the authors identify?
The dataset excludes clinical text and images, and the cohort is under 1% of the source database. Only one foundation model type was evaluated, and generalization to other institutions is uncertain. Some tasks have very few positive labels, making low-variance performance estimates difficult.