# Customize Segment Anything Model for Multi-Modal Semantic Segmentation with Mixture of LoRA Experts Source: [Customize Segment Anything Model for Multi-Modal Semantic Segmentation with Mixture of LoRA Experts]() Collection index: [llms.txt]() Data updated: 2025-10-28T13:57:11.697Z Authors: Chenyang Zhu, Bin Xiao, Lin Shi, Shoukun Xu, Xu Zheng Published: 2024-12-05T14:54:31.000Z ## Abstract The recent Segment Anything Model (SAM) represents a significant breakthrough in scaling segmentation models, delivering strong performance across various downstream applications in the RGB modality. However, directly applying SAM to emerging visual modalities, such as depth and event data results in suboptimal performance in multi-modal segmentation tasks. In this paper, we make the first attempt to adapt SAM for multi-modal semantic segmentation by proposing a Mixture of Low-Rank Adaptation Experts (MoE-LoRA) tailored for different input visual modalities. By training only the MoE-LoRA layers while keeping SAM's weights frozen, SAM's strong generalization and segmentation capabilities can be preserved for downstream tasks. Specifically, to address cross-modal inconsistencies, we propose a novel MoE routing strategy that adaptively generates weighted features across modalities, enhancing multi-modal feature integration. Additionally, we incorporate multi-scale feature extraction and fusion by adapting SAM's segmentation head and introducing an auxiliary segmentation head to combine multi-scale features for improved segmentation performance effectively. Extensive experiments were conducted on three multi-modal benchmarks: DELIVER, MUSES, and MCubeS. The results consistently demonstrate that the proposed method significantly outperforms state-of-the-art approaches across diverse scenarios. Notably, under the particularly challenging condition of missing modalities, our approach exhibits a substantial performance gain, achieving an improvement of 32.15% compared to existing methods. ## Recorded benchmark results A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage. - [Semantic Segmentation on DELIVER RGB (test)](): Semantic segmentation; DELIVER RGB (test); mIoU: 55.23. - [Semantic Segmentation on DELIVER R-D (test)](): Semantic segmentation; DELIVER R-D (test); mIoU: 63.57. - [Semantic Segmentation on DELIVER R-D-E (test)](): Semantic segmentation; DELIVER R-D-E (test); mIoU: 62.69. - [Semantic Segmentation on DELIVER R-D-E-L (test)](): Semantic segmentation; DELIVER R-D-E-L (test); mIoU: 64.08. - [Material Segmentation on MCubeS (test)](): Material Segmentation; MCubeS (test); mIoU: 51.02. ## Original sources and implementation [Original paper]() [PDF]()