# GazeGPT: Augmenting Human Capabilities using Gaze-contingent Contextual AI for Smart Eyewear

Source: [GazeGPT: Augmenting Human Capabilities using Gaze-contingent Contextual AI for Smart Eyewear](<https://www.sota2.com/research/paper/68fa71fd1e209131cea53f25>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2025-10-26T02:16:25.405Z

Authors: Robert Konrad, Nitish Padmanaban, J. Gabriel Buckmaster, Kevin C. Boyle, Gordon Wetzstein

Published: 2024-01-30T18:02:44.000Z

## Abstract

Multimodal large language models (LMMs) excel in world knowledge and problem-solving abilities. Through the use of a world-facing camera and contextual AI, emerging smart accessories aim to provide a seamless interface between humans and LMMs. Yet, these wearable computing systems lack an understanding of the user's attention. We introduce GazeGPT as a new user interaction paradigm for contextual AI. GazeGPT uses eye tracking to help the LMM understand which object in the world-facing camera view a user is paying attention to. Using extensive user evaluations, we show that this gaze-contingent mechanism is a faster and more accurate pointing mechanism than alternatives; that it augments human capabilities by significantly improving their accuracy in a dog-breed classification task; and that it is consistently ranked as more natural than head- or body-driven selection mechanisms for contextual AI. Moreover, we prototype a variety of application scenarios that suggest GazeGPT could be of significant value to users as part of future AI-driven personal assistants.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Cognitive Distraction Detection on DR(eye)VE (Leave-one-dataset-out)](<https://www.sota2.com/research/sota/cognitive-distraction-detection-on-dr-eye-ve-leave-one-dataset-out.md>): Cognitive Distraction Detection; DR(eye)VE (Leave-one-dataset-out); Accuracy: 51.29.

- [Cognitive Distraction Detection on BDD-A (Leave-one-dataset-out)](<https://www.sota2.com/research/sota/cognitive-distraction-detection-on-bdd-a-leave-one-dataset-out.md>): Cognitive Distraction Detection; BDD-A (Leave-one-dataset-out); Accuracy: 51.

- [Cognitive Distraction Detection on DADA 2000 (Leave-one-dataset-out)](<https://www.sota2.com/research/sota/cognitive-distraction-detection-on-dada-2000-leave-one-dataset-out.md>): Cognitive Distraction Detection; DADA 2000 (Leave-one-dataset-out); Accuracy: 51.81.

- [Cognitive Distraction Detection on CogDrive (Aggregated)](<https://www.sota2.com/research/sota/cognitive-distraction-detection-on-cogdrive-aggregated.md>): Cognitive Distraction Detection; CogDrive (Aggregated); Accuracy: 51.69.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2401.17217>)

[PDF](<https://arxiv.org/pdf/2401.17217.pdf>)
