# SAMOT: State-Aware Step Modulation and Optimal Transport Matching for Audio-Visual Instance Segmentation

Source: [SAMOT: State-Aware Step Modulation and Optimal Transport Matching for Audio-Visual Instance Segmentation](<https://www.sota2.com/research/paper/6a7aad9ca080ff033d3e3bb7>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2026-08-12T05:00:05.228Z

Authors: Kai Peng, Yunzhe Shen, Miao Zhang, Leiye Liu, Wei Ji, Jingjing Li, Yongri Piao, Huchuan Lu

Published: 2026-08-08T14:11:56.000Z

## Abstract

Audio-Visual Instance Segmentation (AVIS) aims to simultaneously classify, segment, and track sounding objects within video sequences. Unlike Audio-Visual Semantic Segmentation (AVS), AVIS involves instance-level modeling across longer video sequences, introducing two key challenges: (1) complex modality-state changes disrupt long-range modeling, and (2) substantial structural and distributional discrepancies between modalities hinder precise instance-level association. Existing methods rely on fixed-step Transformers and recursive Mamba models, lacking adaptability to modality-state changes. In addition, methods performing implicit matching ignore the inherent distributional inconsistencies. To address these issues, we propose a framework with Adaptive Dynamic Step Modulation (ADSM) and Optimal Transport-based Matching Modulation (OT-MM). ADSM adaptively modulates Mamba step sizes using temporal variation, cross-modal discrepancy, and historical context, balancing rapid response to modality-state changes with stable long-range modeling. OT-MM explicitly formulates instance-level cross-modal matching as an entropy-regularized optimal transport problem solved via log-domain Sinkhorn iterations, and further enforces distribution-level coherence with an MMD regularizer. Extensive experiments demonstrate state-of-the-art performance on the AVIS benchmark (+3.76 FSLA, +2.75 HOTA, +2.58 mAP), verified through comprehensive qualitative visualizations. The code and model are available at https://github.com/happylife-pk/SAMOT.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2608.08147>)

[PDF](<https://arxiv.org/pdf/2608.08147.pdf>)
