# MoXaRt: Audio-Visual Object-Guided Sound Interaction for XR

Source: [MoXaRt: Audio-Visual Object-Guided Sound Interaction for XR](<https://www.sota2.com/research/paper/69b248a8d030084c22e2f9d9>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2026-03-13T05:00:04.336Z

Authors: Tianyu Xu, Sieun Kim, Qianhui Zheng, Ruoyu Xu, Tejasvi Ravi, Anuva Kulkarni, Katrina Passarella-Ward, Junyi Zhu, Adarsh Kowdle

Published: 2026-03-11T06:35:19.000Z

## Abstract

In Extended Reality (XR), complex acoustic environments often overwhelm users, compromising both scene awareness and social engagement due to entangled sound sources. We introduce MoXaRt, a real-time XR system that uses audio-visual cues to separate these sources and enable fine-grained sound interaction. MoXaRt's core is a cascaded architecture that performs coarse, audio-only separation in parallel with visual detection of sources (e.g., faces, instruments). These visual anchors then guide refinement networks to isolate individual sources, separating complex mixes of up to 5 concurrent sources (e.g., 2 voices + 3 instruments) with ~2 second processing latency. We validate MoXaRt through a technical evaluation on a new dataset of 30 one-minute recordings featuring concurrent speech and music, and a 22-participant user study. Empirical results indicate that our system significantly enhances speech intelligibility, yielding a 36.2% (p \< 0.01) increase in listening comprehension within adversarial acoustic environments while substantially reducing cognitive load (p \< 0.001), thereby paving the way for more perceptive and socially adept XR experiences.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Audio Source Separation on MoXaRt XR Scenarios (human speech sessions)](<https://www.sota2.com/research/sota/audio-source-separation-on-moxart-xr-scenarios-human-speech-sessions.md>): Audio Source Separation; MoXaRt XR Scenarios (human speech sessions); WER: 0.3824.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2603.10465>)

[PDF](<https://arxiv.org/pdf/2603.10465.pdf>)
