# FHIRPath-QA: Executable Question Answering over FHIR Electronic Health Records Source: [FHIRPath-QA: Executable Question Answering over FHIR Electronic Health Records]() Collection index: [llms.txt]() Data updated: 2026-03-27T05:00:04.193Z Authors: Michael Frew, Nishit Bheda, Bryan Tripp Published: 2026-02-26T20:14:21.000Z ## Abstract Though patients are increasingly granted digital access to their electronic health records (EHRs), existing interfaces may not support precise, trustworthy answers to patient-specific questions. Large language models (LLM) show promise in clinical question answering (QA), but retrieval-based approaches are computationally inefficient, prone to hallucination, and difficult to deploy over real-life EHRs. This work introduces FHIRPath-QA, the first open dataset and benchmark for patient-specific QA that includes open-standard FHIRPath queries over real-world clinical data. A text-to-FHIRPath QA paradigm is proposed that shifts reasoning from free-text generation to FHIRPath query synthesis. For o4-mini, this reduced average token usage by 391x relative to retrieval-first prompting (629,829 vs 1,609 tokens per question) and lowered failure rates from 0.36 to 0.09 on clinician-phrased questions. Built on MIMIC-IV on FHIR Demo, the dataset pairs over 14k natural language questions in patient and clinician phrasing with validated FHIRPath queries and answers. Empirically, the evaluated LLMs achieve at most 42% accuracy, highlighting the challenge of the task, but benefit strongly from supervised fine-tuning, with query synthesis accuracy improving from 27% to 79% for 4o-mini. These results highlight that text-to-FHIRPath synthesis has the potential to serve as a practical foundation for safe, efficient, and interoperable consumer health applications, and the FHIRPath-QA dataset and benchmark serve as a starting point for future research on the topic. The full dataset and generation code can be accessed at: https://github.com/mooshifrew/fhirpath-qa. ## Recorded benchmark results A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage. - [Natural Language to FHIRPath query generation on FHIRPath-QA Clinical perspective](): Natural Language to FHIRPath query generation; FHIRPath-QA Clinical perspective; Accuracy: not recorded. - [Natural Language to FHIRPath query generation on FHIRPath-QA Patient perspective](): Natural Language to FHIRPath query generation; FHIRPath-QA Patient perspective; Accuracy: not recorded. ## Original sources and implementation [Original paper]() [PDF]()