Towards Generalist Foundation Model for Radiology by Leveraging Web-scale 2D&3D Medical Data
About
In this study, we aim to initiate the development of Radiology Foundation Model, termed as RadFM. We consider the construction of foundational models from three perspectives, namely, dataset construction, model design, and thorough evaluation. Our contribution can be concluded as follows: (i), we construct a large-scale Medical Multi-modal Dataset, MedMD, which consists of 16M 2D and 3D medical scans with high-quality text descriptions or reports across various data formats, modalities, and tasks, covering over 5000 distinct diseases. To the best of our knowledge, this is the first large-scale, high-quality, medical visual-language dataset, with both 2D and 3D scans; (ii), we propose an architecture that enables visually conditioned generative pre-training, i.e., allowing for integration of text input with 2D or 3D medical scans, and generate responses for diverse radiologic tasks. The model was initially pre-trained on MedMD and subsequently fine-tuned on the domain-specific dataset, which is a radiologic cleaned version of MedMD, containing 3M radiologic visual-language pairs, termed as RadMD; (iii), we propose a new evaluation benchmark, RadBench, that comprises five tasks, including modality recognition, disease diagnosis, visual question answering, report generation and rationale diagnosis, aiming to comprehensively assess the capability of foundation models in handling practical clinical problems. We conduct both automatic and human evaluation on RadBench, in both cases, RadFM outperforms existing multi-modal foundation models, that are publicaly accessible, including Openflamingo, MedFlamingo, MedVInT and GPT-4V. Additionally, we also adapt RadFM for different public benchmarks, surpassing existing SOTAs on diverse datasets. All codes, data, and model checkpoint will all be made publicly available to promote further research and development in the field.
Key insights
What problem does RadFM address and how?
RadFM tackles the lack of generalist radiology models that handle both 2D and 3D medical scans. It uses a 3D Vision Transformer and a perceiver module to fuse visual scans with text, generating responses for diverse clinical tasks via autoregressive pre-training on 16M image-text pairs.
How does it differ from prior medical models?
Unlike prior models limited to 2D images, RadFM unifies 2D and 3D inputs. On the RadBench benchmark, it outperforms OpenFlamingo, MedVInT, Med-Flamingo, and GPT-4V across five tasks including disease diagnosis and report generation.
What are the practical uses of these findings?
RadFM is a research prototype demonstrating potential for clinical tasks like disease diagnosis, visual question answering, and report generation. The paper does not document any actual clinical deployment; it establishes a benchmark and model for future development.
What limitations do the authors acknowledge?
The authors note that long sentence generation remains underdeveloped, 3D images are limited in the dataset, automatic metrics fall short of capturing medical precision, and missing 3D metadata prevents precise size measurements in reports.