# GazeLLM: Multimodal LLMs incorporating Human Visual Attention

Source: [GazeLLM: Multimodal LLMs incorporating Human Visual Attention](<https://www.sota2.com/research/paper/68fa77d91e209131cea7587c>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2025-10-27T06:50:54.619Z

Authors: Jun Rekimoto

Published: 2025-03-31T20:50:04.000Z

## Abstract

Large Language Models (LLMs) are advancing into Multimodal LLMs (MLLMs), capable of processing image, audio, and video as well as text. Combining first-person video, MLLMs show promising potential for understanding human activities through video and audio, enabling many human-computer interaction and human-augmentation applications such as human activity support, real-world agents, and skill transfer to robots or other individuals. However, handling high-resolution, long-duration videos generates large latent representations, leading to substantial memory and processing demands, limiting the length and resolution MLLMs can manage. Reducing video resolution can lower memory usage but often compromises comprehension. This paper introduces a method that optimizes first-person video analysis by integrating eye-tracking data, and proposes a method that decomposes first-person vision video into sub areas for regions of gaze focus. By processing these selectively gazed-focused inputs, our approach achieves task comprehension equivalent to or even better than processing the entire image at full resolution, but with significantly reduced video data input (reduce the number of pixels to one-tenth), offering an efficient solution for using MLLMs to interpret and utilize human skills.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Cognitive Distraction Detection on BDD-A (Leave-one-dataset-out)](<https://www.sota2.com/research/sota/cognitive-distraction-detection-on-bdd-a-leave-one-dataset-out.md>): Cognitive Distraction Detection; BDD-A (Leave-one-dataset-out); Accuracy: 44.53.

- [Cognitive Distraction Detection on DADA 2000 (Leave-one-dataset-out)](<https://www.sota2.com/research/sota/cognitive-distraction-detection-on-dada-2000-leave-one-dataset-out.md>): Cognitive Distraction Detection; DADA 2000 (Leave-one-dataset-out); Accuracy: 49.09.

- [Cognitive Distraction Detection on DR(eye)VE (Leave-one-dataset-out)](<https://www.sota2.com/research/sota/cognitive-distraction-detection-on-dr-eye-ve-leave-one-dataset-out.md>): Cognitive Distraction Detection; DR(eye)VE (Leave-one-dataset-out); Accuracy: 49.14.

- [Cognitive Distraction Detection on CogDrive (Aggregated)](<https://www.sota2.com/research/sota/cognitive-distraction-detection-on-cogdrive-aggregated.md>): Cognitive Distraction Detection; CogDrive (Aggregated); Accuracy: 48.35.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2504.00221>)

[PDF](<https://arxiv.org/pdf/2504.00221.pdf>)
