# Adapting 2D Multi-Modal Large Language Model for 3D CT Image Analysis Source: [Adapting 2D Multi-Modal Large Language Model for 3D CT Image Analysis]() Collection index: [llms.txt]() Data updated: 2026-04-15T05:00:04.591Z Authors: Yang Yu, Dunyuan Xu, Yaoqian Li, Xiaomeng Li, Jinpeng Li, Pheng-Ann Heng Published: 2026-04-11T14:36:05.000Z ## Abstract 3D medical image analysis is of great importance in disease diagnosis and treatment. Recently, multimodal large language models (MLLMs) have exhibited robust perceptual capacity, strong cross-modal alignment, and promising generalizability. Therefore, they have great potential to improve the performance of medical report generation (MRG) and medical visual question answering (MVQA), which serve as two important tasks in clinical scenarios. However, due to the scarcity of 3D medical images, existing 3D medical MLLMs suffer from insufficiently pretrained vision encoder and inability to extract customized image features for different kinds of tasks. In this paper, we propose to first transfer a 2D MLLM, which is well trained with 2D natural images, to support 3D medical volumetric inputs while reusing all of its pre-trained parameters. To enable the vision encoder to extract tailored image features for various tasks, we then design a Text-Guided Hierarchical MoE (TGH-MoE) framework, which can distinguish tasks under the guidance of the text prompt. Furthermore, we propose a two-stage training strategy to learn both task-shared and task-specific image features. As demonstrated empirically, our method outperforms existing 3D medical MLLMs in both MRG and MVQA tasks. Our code will be released once this paper is accepted. ## Recorded benchmark results A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage. - [Medical Visual Question Answering on M3D-Data (test)](): Medical Visual Question Answering; M3D-Data (test); BLEU-1: 54.64. - [Medical Report Generation on M3D-Data (test)](): Medical Report Generation; M3D-Data (test); BLEU-1: 14.23. ## Original sources and implementation [Original paper]() [PDF]()