# LatentOmni: Rethinking Omni-Modal Understanding via Unified Audio-Visual Latent Reasoning

Source: [LatentOmni: Rethinking Omni-Modal Understanding via Unified Audio-Visual Latent Reasoning](<https://www.sota2.com/research/paper/6a0fe328f984739ed6739ea8>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2026-05-25T06:00:00.890Z

Authors: Yifan Dai, Zhenhua Wu, Bohan Zeng, Daili Hua, Jialing Liu, Bozhou Li, Yuran Wang, Chengzhuo Tong, Hao Liang, Xiaochen Ma, Junbo Niu, Tianyu Guo, Yang Shi, Yue Ding, Yiyan Ji, Bingyin Mei, Yushuo Guan, Yuanxing Zhang, Pengfei Wan, Fangcheng Fu, Wentao Zhang

Published: 2026-05-21T05:18:57.000Z

## Abstract

Joint audio-visual reasoning is essential for omnimodal understanding, yet current multimodal large language models (MLLMs) still struggle when reasoning requires fine-grained evidence from both modalities. A central limitation is that explicit text-based chain-of-thought (CoT) compresses continuous audio-visual signals into discrete tokens, weakening temporal grounding and shifting intermediate reasoning toward language priors. We argue that a unified latent space is a better medium for such reasoning because it preserves dense sensory information while remaining compatible with autoregressive generation. Based on this insight, we propose \\textbf{LatentOmni}, a cross-modal reasoning framework that interleaves textual reasoning with audio-visual latent states. LatentOmni introduces feature-level supervision to align latent reasoning states with task-relevant sensory features and uses Omni-Sync Position Embedding (OSPE) to maintain temporal consistency between latent audio and visual states. We further construct \\textbf{LatentOmni-Instruct-35K}, a dataset of audio-visual interleaved reasoning trajectories for supervising latent-space reasoning. Comprehensive evaluation across multiple audio-visual reasoning benchmarks demonstrates that LatentOmni achieves the best performance among the evaluated open-source models and consistently outperforms the Explicit Text CoT baseline, supporting latent-space joint reasoning as a promising path toward stronger omnimodal understanding.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Long-video sustained reasoning on LVOmniBench](<https://www.sota2.com/research/sota/long-video-sustained-reasoning-on-lvomnibench.md>): Long-video sustained reasoning; LVOmniBench; Accuracy: 35.1.

- [Common Sense Reasoning on WorldSense](<https://www.sota2.com/research/sota/common-sense-reasoning-on-worldsense.md>): Common Sense Reasoning; WorldSense; Accuracy: 48.9.

- [Omnimodal common event understanding on Daily-Omni](<https://www.sota2.com/research/sota/omnimodal-common-event-understanding-on-daily-omni.md>): Omnimodal common event understanding; Daily-Omni; Accuracy: 67.4.

- [Fine-grained audio-visual video understanding on OmniVideoBench](<https://www.sota2.com/research/sota/fine-grained-audio-visual-video-understanding-on-omnivideobench.md>): Fine-grained audio-visual video understanding; OmniVideoBench; Accuracy: 35.4.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2605.22012>)

[PDF](<https://arxiv.org/pdf/2605.22012.pdf>)

[Implementation](<https://github.com/yfanDai/LatentOmni>)