# AURORA:Augmented Understanding via Structured Reasoning and Reinforcement Learning for Reference Audio-Visual Segmentation

Source: [AURORA:Augmented Understanding via Structured Reasoning and Reinforcement Learning for Reference Audio-Visual Segmentation](<https://www.sota2.com/research/paper/68fa7a6a1e209131cea830e5>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2025-12-12T05:01:30.602Z

Authors: Ziyang Luo, Nian Liu, Fahad Shahbaz Khan, Junwei Han

Published: 2025-08-04T07:47:38.000Z

## Abstract

Reference Audio-Visual Segmentation (Ref-AVS) tasks challenge models to precisely locate sounding objects by integrating visual, auditory, and textual cues. Existing methods often lack genuine semantic understanding, tending to memorize fixed reasoning patterns. Furthermore, jointly training for reasoning and segmentation can compromise pixel-level precision. To address these issues, we introduce AURORA, a novel framework designed to enhance genuine reasoning and language comprehension in reference audio-visual segmentation. We employ a structured Chain-of-Thought (CoT) prompting mechanism to guide the model through a step-by-step reasoning process and introduce a novel segmentation feature distillation loss to effectively integrate these reasoning abilities without sacrificing segmentation performance. To further cultivate the model's genuine reasoning capabilities, we devise a further two-stage training strategy: first, a \`\`corrective reflective-style training" stage utilizes self-correction to enhance the quality of reasoning paths, followed by reinforcement learning via Group Reward Policy Optimization (GRPO) to bolster robustness in challenging scenarios. Experiments demonstrate that AURORA achieves state-of-the-art performance on Ref-AVS benchmarks and generalizes effectively to unreferenced segmentation.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Referential Audio-Visual Segmentation on Ref-AVS (seen)](<https://www.sota2.com/research/sota/referential-audio-visual-segmentation-on-ref-avs-seen.md>): Referential Audio-Visual Segmentation; Ref-AVS (seen); J & F Score: 0.68.

- [Referring Audio-Visual Segmentation on Ref-AVS (mix)](<https://www.sota2.com/research/sota/referring-audio-visual-segmentation-on-ref-avs-mix.md>): Referring Audio-Visual Segmentation; Ref-AVS (mix); Jaccard Index (J): 66.5.

- [Referring Audio-Visual Segmentation on Ref-AVS (unseen)](<https://www.sota2.com/research/sota/referring-audio-visual-segmentation-on-ref-avs-unseen.md>): Referring Audio-Visual Segmentation; Ref-AVS (unseen); Jaccard Index (J): 69.7.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2508.02149>)

[PDF](<https://arxiv.org/pdf/2508.02149.pdf>)