# Intra-model Variability in COVID-19 Classification Using Chest X-ray Images

Source: [Intra-model Variability in COVID-19 Classification Using Chest X-ray Images](<https://www.sota2.com/research/paper/68fa677a1e209131cea16a3b>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2025-10-25T22:16:13.371Z

Authors: Brian D Goodwin, Corey Jaskolski, Can Zhong, Herick Asmani

Published: 2020-04-30T21:20:32.000Z

## Abstract

X-ray and computed tomography (CT) scanning technologies for COVID-19 screening have gained significant traction in AI research since the start of the coronavirus pandemic. Despite these continuous advancements for COVID-19 screening, many concerns remain about model reliability when used in a clinical setting. Much has been published, but with limited transparency in expected model performance. We set out to address this limitation through a set of experiments to quantify baseline performance metrics and variability for COVID-19 detection in chest x-ray for 12 common deep learning architectures. Specifically, we adopted an experimental paradigm controlling for train-validation-test split and model architecture where the source of prediction variability originates from model weight initialization, random data augmentation transformations, and batch shuffling. Each model architecture was trained 5 separate times on identical train-validation-test splits of a publicly available x-ray image dataset provided by Cohen et al. (2020). Results indicate that even within model architectures, model behavior varies in a meaningful way between trained models. Best performing models achieve a false negative rate of 3 out of 20 for detecting COVID-19 in a hold-out set. While these results show promise in using AI for COVID-19 screening, they further support the urgent need for diverse medical imaging datasets for model training in a way that yields consistent prediction outcomes. It is our hope that these modeling results accelerate work in building a more robust dataset and a viable screening tool for COVID-19.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Pneumonia Classification on Cohen](<https://www.sota2.com/research/sota/pneumonia-classification-on-cohen.md>): Pneumonia Classification; Cohen; Accuracy: 89.4.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2005.02167>)

[PDF](<https://arxiv.org/pdf/2005.02167.pdf>)
