# Hallucination Filtering in Radiology Vision-Language Models Using Discrete Semantic Entropy

Source: [Hallucination Filtering in Radiology Vision-Language Models Using Discrete Semantic Entropy](<https://www.sota2.com/research/paper/68fa7bff1e209131cea8bc97>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2026-02-27T05:00:04.201Z

Authors: Patrick Wienholt, Sophie Caselitz, Robert Siepmann, Philipp Bruners, Keno Bressem, Christiane Kuhl, Jakob Nikolas Kather, Sven Nebelung, Daniel Truhn

Published: 2025-10-10T10:53:33.000Z

## Abstract

To determine whether using discrete semantic entropy (DSE) to reject questions likely to generate hallucinations can improve the accuracy of black-box vision-language models (VLMs) in radiologic image based visual question answering (VQA). This retrospective study evaluated DSE using two publicly available, de-identified datasets: the VQA-Med 2019 benchmark (500 images with clinical questions and short-text answers) and a diagnostic radiology dataset (206 cases: 60 computed tomography scans, 60 magnetic resonance images, 60 radiographs, 26 angiograms) with corresponding ground-truth diagnoses. GPT-4o and GPT-4.1 (Generative Pretrained Transformer; OpenAI) answered each question 15 times using a temperature of 1.0. Baseline accuracy was determined using low-temperature answers (temperature 0.1). Meaning-equivalent responses were grouped using bidirectional entailment checks, and DSE was computed from the relative frequencies of the resulting semantic clusters. Accuracy was recalculated after excluding questions with DSE \> 0.6 or \> 0.3. p-values and 95% confidence intervals were obtained using bootstrap resampling and a Bonferroni-corrected threshold of p \< .004 for statistical significance. Across 706 image-question pairs, baseline accuracy was 51.7% for GPT-4o and 54.8% for GPT-4.1. After filtering out high-entropy questions (DSE \> 0.3), accuracy on the remaining questions was 76.3% (retained questions: 334/706) for GPT-4o and 63.8% (retained questions: 499/706) for GPT-4.1 (both p \< .001). Accuracy gains were observed across both datasets and largely remained statistically significant after Bonferroni correction. DSE enables reliable hallucination detection in black-box VLMs by quantifying semantic inconsistency. This method significantly improves diagnostic answer accuracy and offers a filtering strategy for clinical VLM applications.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Diagnostic Classification on derm7pt Melanoma vs. Atypical Nevus (test)](<https://www.sota2.com/research/sota/diagnostic-classification-on-derm7pt-melanoma-vs-atypical-nevus-test.md>): Diagnostic Classification; derm7pt Melanoma vs. Atypical Nevus (test); Accuracy: 71.9.

- [Diagnostic Classification on MIMIC-CXR Edema vs. Pneumonia (test)](<https://www.sota2.com/research/sota/diagnostic-classification-on-mimic-cxr-edema-vs-pneumonia-test.md>): Diagnostic Classification; MIMIC-CXR Edema vs. Pneumonia (test); Accuracy: 61.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2510.09256>)

[PDF](<https://arxiv.org/pdf/2510.09256.pdf>)
