# Voila-A: Aligning Vision-Language Models with User's Gaze Attention

Source: [Voila-A: Aligning Vision-Language Models with User's Gaze Attention](<https://www.sota2.com/research/paper/68fa71e11e209131cea53885>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2025-10-26T02:14:27.865Z

Authors: Kun Yan, Lei Ji, Zeyu Wang, Yuntao Wang, Nan Duan, Shuai Ma

Published: 2023-12-22T17:34:01.000Z

## Abstract

In recent years, the integration of vision and language understanding has led to significant advancements in artificial intelligence, particularly through Vision-Language Models (VLMs). However, existing VLMs face challenges in handling real-world applications with complex scenes and multiple objects, as well as aligning their focus with the diverse attention patterns of human users. In this paper, we introduce gaze information, feasibly collected by AR or VR devices, as a proxy for human attention to guide VLMs and propose a novel approach, Voila-A, for gaze alignment to enhance the interpretability and effectiveness of these models in real-world applications. First, we collect hundreds of minutes of gaze data to demonstrate that we can mimic human gaze modalities using localized narratives. We then design an automatic data annotation pipeline utilizing GPT-4 to generate the VOILA-COCO dataset. Additionally, we innovate the Voila Perceiver modules to integrate gaze information into VLMs while preserving their pretrained knowledge. We evaluate Voila-A using a hold-out validation set and a newly collected VOILA-GAZE Testset, which features real-life scenarios captured with a gaze-tracking device. Our experimental results demonstrate that Voila-A significantly outperforms several baseline models. By aligning model attention with human gaze patterns, Voila-A paves the way for more intuitive, user-centric VLMs and fosters engaging human-AI interaction across a wide range of applications.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Cognitive Distraction Detection on DADA 2000 (Leave-one-dataset-out)](<https://www.sota2.com/research/sota/cognitive-distraction-detection-on-dada-2000-leave-one-dataset-out.md>): Cognitive Distraction Detection; DADA 2000 (Leave-one-dataset-out); Accuracy: 67.68.

- [Cognitive Distraction Detection on BDD-A (Leave-one-dataset-out)](<https://www.sota2.com/research/sota/cognitive-distraction-detection-on-bdd-a-leave-one-dataset-out.md>): Cognitive Distraction Detection; BDD-A (Leave-one-dataset-out); Accuracy: 58.81.

- [Cognitive Distraction Detection on DR(eye)VE (Leave-one-dataset-out)](<https://www.sota2.com/research/sota/cognitive-distraction-detection-on-dr-eye-ve-leave-one-dataset-out.md>): Cognitive Distraction Detection; DR(eye)VE (Leave-one-dataset-out); Accuracy: 54.91.

- [Cognitive Distraction Detection on CogDrive (Aggregated)](<https://www.sota2.com/research/sota/cognitive-distraction-detection-on-cogdrive-aggregated.md>): Cognitive Distraction Detection; CogDrive (Aggregated); Accuracy: 62.81.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2401.09454>)

[PDF](<https://arxiv.org/pdf/2401.09454.pdf>)
