# OmniVinci: Enhancing Architecture and Data for Omni-Modal Understanding LLM

Source: [OmniVinci: Enhancing Architecture and Data for Omni-Modal Understanding LLM](<https://www.sota2.com/research/paper/68fa7c481e209131cea8d652>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2025-10-30T05:01:12.438Z

Authors: Hanrong Ye, Chao-Han Huck Yang, Arushi Goel, Wei Huang, Ligeng Zhu, Yuanhang Su, Sean Lin, An-Chieh Cheng, Zhen Wan, Jinchuan Tian, Yuming Lou, Dong Yang, Zhijian Liu, Yukang Chen, Ambrish Dantrey, Ehsan Jahangiri, Sreyan Ghosh, Daguang Xu, Ehsan Hosseini-Asl, Danial Mohseni Taheri, Vidya Murali, Sifei Liu, Yao Lu, Oluwatobi Olabiyi, Yu-Chiang Frank Wang, Rafael Valle, Bryan Catanzaro, Andrew Tao, Song Han, Jan Kautz, Hongxu Yin, Pavlo Molchanov

Published: 2025-10-17T17:59:59.000Z

## Abstract

Advancing machine intelligence requires developing the ability to perceive across multiple modalities, much as humans sense the world. We introduce OmniVinci, an initiative to build a strong, open-source, omni-modal LLM. We carefully study the design choices across model architecture and data curation. For model architecture, we present three key innovations: (i) OmniAlignNet for strengthening alignment between vision and audio embeddings in a shared omni-modal latent space; (ii) Temporal Embedding Grouping for capturing relative temporal alignment between vision and audio signals; and (iii) Constrained Rotary Time Embedding for encoding absolute temporal information in omni-modal embeddings. We introduce a curation and synthesis pipeline that generates 24M single-modal and omni-modal conversations. We find that modalities reinforce one another in both perception and reasoning. Our model, OmniVinci, outperforms Qwen2.5-Omni with +19.05 on DailyOmni (cross-modal understanding), +1.7 on MMAR (audio), and +3.9 on Video-MME (vision), while using just 0.2T training tokens - a 6 times reduction compared to Qwen2.5-Omni's 1.2T. We finally demonstrate omni-modal advantages in downstream applications spanning robotics, medical AI, and smart factory.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Omnimodal Video-Audio Question Answering on IMAVB 1.0](<https://www.sota2.com/research/sota/omnimodal-video-audio-question-answering-on-imavb-1-0.md>): Omnimodal Video-Audio Question Answering; IMAVB 1.0; Std Dev Score (Video): 75.4.

- [Social Cues Attribution on Ego4D](<https://www.sota2.com/research/sota/social-cues-attribution-on-ego4d.md>): Social Cues Attribution; Ego4D; Verbal Score: 54.04.

- [Social Cues Attribution on YouTube](<https://www.sota2.com/research/sota/social-cues-attribution-on-youtube.md>): Social Cues Attribution; YouTube; Verbal Score: 58.1.

- [Audio-Visual Understanding on Video-MME](<https://www.sota2.com/research/sota/audio-visual-understanding-on-video-mme.md>): Audio-visual understanding; Video-MME; Score: 68.6.

- [Audiovisual Understanding & Reasoning on World-Sense](<https://www.sota2.com/research/sota/audiovisual-understanding-and-reasoning-on-world-sense.md>): Audiovisual Understanding & Reasoning; World-Sense; Score: 48.2.

- [Coreference Alignment on CrossOmni](<https://www.sota2.com/research/sota/coreference-alignment-on-crossomni.md>): Coreference Alignment; CrossOmni; Au Score: 51.76.

- [Audio-Visual Video Understanding on Video-MME truncated \<= 5min v2](<https://www.sota2.com/research/sota/audio-visual-video-understanding-on-video-mme-truncated-5min-v2.md>): Audio-Visual Video Understanding; Video-MME truncated \<= 5min v2; Overall Score: 11.59.

- [Audio-Visual Video Understanding on JointAVBench truncated \<= 5min](<https://www.sota2.com/research/sota/audio-visual-video-understanding-on-jointavbench-truncated-5min.md>): Audio-Visual Video Understanding; JointAVBench truncated \<= 5min; Overall Score: 57.55.

- [Audio-Visual Video Understanding on OmniVideoBench truncated \<= 5min](<https://www.sota2.com/research/sota/audio-visual-video-understanding-on-omnivideobench-truncated-5min.md>): Audio-Visual Video Understanding; OmniVideoBench truncated \<= 5min; Score: 52.81.

- [Audio-Visual Video Understanding on Video-MME short](<https://www.sota2.com/research/sota/audio-visual-video-understanding-on-video-mme-short.md>): Audio-Visual Video Understanding; Video-MME Short; Overall Score: 77.56.

- [Multimodal Perception on MME Perception](<https://www.sota2.com/research/sota/multimodal-perception-on-mme-perception.md>): Multimodal Perception; MME Perception; Perception Score: 1651.

- [Multi-Modal Social Interaction on Werewolf Among Us (YouTube)](<https://www.sota2.com/research/sota/multi-modal-social-interaction-on-werewolf-among-us-youtube.md>): Multi-Modal Social Interaction; Werewolf Among Us (YouTube); STI: 19.57.

- [Social Understanding on OmniSocialBench](<https://www.sota2.com/research/sota/social-understanding-on-omnisocialbench.md>): Social Understanding; OmniSocialBench; Social Norm Accuracy: 18.75.

- [Multi-Modal Social Interaction on Werewolf Among Us Ego4D](<https://www.sota2.com/research/sota/multi-modal-social-interaction-on-werewolf-among-us-ego4d.md>): Multi-Modal Social Interaction; Werewolf Among Us Ego4D; STI: 24.57.

- [Time-aware Dense Captioning on OmniDCBench 1.0 (test)](<https://www.sota2.com/research/sota/time-aware-dense-captioning-on-omnidcbench-1-0-test.md>): Time-aware Dense Captioning; OmniDCBench 1.0 (test); Camera Score: 1.6.

- [Cross-modal Hallucination Detection on Curse of Multi-Modalities (CMM) 1.0 (test)](<https://www.sota2.com/research/sota/cross-modal-hallucination-detection-on-curse-of-multi-modalities-cmm-1-0-test.md>): Cross-modal Hallucination Detection; Curse of Multi-Modalities (CMM) 1.0 (test); VL Precision (pa): 88.5.

- [Video Question Answering on OmniVideo (test)](<https://www.sota2.com/research/sota/video-question-answering-on-omnivideo-test.md>): Video Question Answering; OmniVideo (test); Overall Score: 47.13.

- [Multi-Scene Segmentation on OmniDCBench 1.0 (test)](<https://www.sota2.com/research/sota/multi-scene-segmentation-on-omnidcbench-1-0-test.md>): Multi-Scene Segmentation; OmniDCBench 1.0 (test); F1 Score: 29.

- [Audiovisual Understanding & Reasoning on Daily-Omni](<https://www.sota2.com/research/sota/audiovisual-understanding-and-reasoning-on-daily-omni.md>): Audiovisual Understanding & Reasoning; Daily-Omni; Score: 66.5.

- [Audiovisual Matching on AVHBench](<https://www.sota2.com/research/sota/audiovisual-matching-on-avhbench.md>): Audiovisual Matching; AVHBench; Accuracy: 54.32.

- [Audiovisual Dialogue Description on DiaDemBench](<https://www.sota2.com/research/sota/audiovisual-dialogue-description-on-diadembench.md>): Audiovisual Dialogue Description; DiaDemBench; REF: 17.1.

- [Audio-Visual Understanding on DailyOmni](<https://www.sota2.com/research/sota/audio-visual-understanding-on-dailyomni.md>): Audio-visual understanding; DailyOmni; Average Score: 66.5.

- [Cross-modal hallucination evaluation on AVHBench](<https://www.sota2.com/research/sota/cross-modal-hallucination-evaluation-on-avhbench.md>): Cross-modal hallucination evaluation; AVHBench; Overall Accuracy: 61.36.

- [Video-driven Audio Hallucination on AVHBench](<https://www.sota2.com/research/sota/video-driven-audio-hallucination-on-avhbench.md>): Video-driven Audio Hallucination; AVHBench; Accuracy: 58.56.

- [Object Hallucination Evaluation on POPE](<https://www.sota2.com/research/sota/object-hallucination-evaluation-on-pope.md>): Object Hallucination Evaluation; POPE; Accuracy: 89.5.

- [Audio-visual understanding on WorldSense](<https://www.sota2.com/research/sota/audio-visual-understanding-on-worldsense.md>): Audio-visual understanding; WorldSense; Accuracy: 48.2.

- [Multimodal Understanding on MMMU](<https://www.sota2.com/research/sota/multimodal-understanding-on-mmmu.md>): Multimodal Understanding; MMMU; Accuracy: 49.7.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2510.15870>)

[PDF](<https://arxiv.org/pdf/2510.15870.pdf>)
