# Tracking Listener Attention: Gaze-Guided Audio-Visual Speech Enhancement Framework

Source: [Tracking Listener Attention: Gaze-Guided Audio-Visual Speech Enhancement Framework](<https://www.sota2.com/research/paper/69d8843a7d0172140ec77bfc>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2026-04-11T05:00:11.573Z

Authors: Hsiang-Cheng Yang, You-Jin Li, Rong Chao, Yu Tsao, Borching Su, Shao-Yi Chien

Published: 2026-04-09T15:22:50.000Z

## Abstract

This paper presents a Gaze-Guided Audio-Visual Speech Enhancement (GG-AVSE) framework to address the cocktail party problem. A major challenge in conventional AVSE is identifying the listener's intended speaker in multi-talker environments. GG-AVSE addresses this issue by exploiting gaze direction as a supervisory cue for target-speaker selection. Specifically, we propose the GG-VM module, which combines gaze signals with a YOLO5Face detector to extract the target speaker's facial features and integrates them with the pretrained AVSEMamba model through two strategies: zero-shot merging and partial visual fine-tuning. For evaluation, we introduce the AVSEC2-Gaze dataset. Experimental results show that GG-AVSE achieves substantial performance gains over gaze-free baselines: a 10.08% improvement in PESQ (2.370 to 2.609), a 5.18% improvement in STOI (0.8802 to 0.9258), and a 23.69% improvement in SI-SDR (9.16 to 11.33). These results confirm that gaze provides an effective cue for resolving target-speaker ambiguity and highlight the scalability of GG-AVSE for real-world applications.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Speech Enhancement on AVSEC2-Gaze](<https://www.sota2.com/research/sota/speech-enhancement-on-avsec2-gaze.md>): Speech Enhancement; AVSEC2-Gaze; PESQ: 2.609.

- [Speech Enhancement on AVSEC-2](<https://www.sota2.com/research/sota/speech-enhancement-on-avsec-2.md>): Speech Enhancement; AVSEC-2; PESQ: not recorded.

- [Speech Enhancement on AVSEC-2 200 (test)](<https://www.sota2.com/research/sota/speech-enhancement-on-avsec-2-200-test.md>): Speech Enhancement; AVSEC-2 200 (test); PESQ: not recorded.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2604.08359>)

[PDF](<https://arxiv.org/pdf/2604.08359.pdf>)