# MM-OVSeg:Multimodal Optical-SAR Fusion for Open-Vocabulary Segmentation in Remote Sensing

Source: [MM-OVSeg:Multimodal Optical-SAR Fusion for Open-Vocabulary Segmentation in Remote Sensing](<https://www.sota2.com/research/paper/69bb83230636e5bf4afc1b8d>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2026-03-31T05:00:04.048Z

Authors: Yimin Wei, Aoran Xiao, Hongruixuan Chen, Junshi Xia, Naoto Yokoya

Published: 2026-03-18T09:34:23.000Z

## Abstract

Open-vocabulary segmentation enables pixel-level recognition from an open set of textual categories, allowing generalization beyond fixed classes. Despite great potential in remote sensing, progress in this area remains largely limited to clear-sky optical data and struggles under cloudy or haze-contaminated conditions. We present MM-OVSeg, a multimodal Optical-SAR fusion framework for resilient open-vocabulary segmentation under adverse weather conditions. MM-OVSeg leverages the complementary strengths of the two modalities--optical imagery provides rich spectral semantics, while synthetic aperture radar (SAR) offers cloud-penetrating structural cues. To address the cross-modal domain gap and the limited dense prediction capability of current vision-language models, we propose two key designs: a cross-modal unification process for multi-sensor representation alignment, and a dual-encoder fusion module that integrates hierarchical features from multiple vision foundation models for text-aligned multimodal segmentation. Extensive experiments demonstrate that MM-OVSeg achieves superior robustness and generalization across diverse cloud conditions. The source dataset and code are available at https://github.com/Jimmyxichen/MM-OVSeg.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Open-Vocabulary Semantic Segmentation on PIE-RGB-SAR cloud](<https://www.sota2.com/research/sota/open-vocabulary-semantic-segmentation-on-pie-rgb-sar-cloud.md>): Open Vocabulary Semantic Segmentation; PIE-RGB-SAR cloud; mIoU: 57.7.

- [Open-Vocabulary Semantic Segmentation on DDHR-SK](<https://www.sota2.com/research/sota/open-vocabulary-semantic-segmentation-on-ddhr-sk.md>): Open Vocabulary Semantic Segmentation; DDHR-SK; mIoU: 73.1.

- [Open-Vocabulary Semantic Segmentation on OpenEarthMap-SAR thick](<https://www.sota2.com/research/sota/open-vocabulary-semantic-segmentation-on-openearthmap-sar-thick.md>): Open Vocabulary Semantic Segmentation; OpenEarthMap-SAR thick; mIoU: 36.6.

- [Open-Vocabulary Semantic Segmentation on OpenEarthMap-SAR OEM-thin](<https://www.sota2.com/research/sota/open-vocabulary-semantic-segmentation-on-openearthmap-sar-oem-thin.md>): Open Vocabulary Semantic Segmentation; OpenEarthMap-SAR OEM-thin; mIoU: 40.2.

- [Open-Vocabulary Semantic Segmentation on PIE-RGB-SAR clean](<https://www.sota2.com/research/sota/open-vocabulary-semantic-segmentation-on-pie-rgb-sar-clean.md>): Open Vocabulary Semantic Segmentation; PIE-RGB-SAR clean; mIoU: 59.7.

- [Open-Vocabulary Semantic Segmentation on DDHR-SK to DDHR-CH](<https://www.sota2.com/research/sota/open-vocabulary-semantic-segmentation-on-ddhr-sk-to-ddhr-ch.md>): Open Vocabulary Semantic Segmentation; DDHR-SK to DDHR-CH; mIoU: 42.6.

- [Open-Vocabulary Semantic Segmentation on MM-OVSeg Evaluation Suite Combined](<https://www.sota2.com/research/sota/open-vocabulary-semantic-segmentation-on-mm-ovseg-evaluation-suite-combined.md>): Open Vocabulary Semantic Segmentation; MM-OVSeg Evaluation Suite Combined; mIoU: 51.7.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2603.17528>)

[PDF](<https://arxiv.org/pdf/2603.17528.pdf>)