# Training Computer Use Agents to Assess the Usability of Graphical User Interfaces

Source: [Training Computer Use Agents to Assess the Usability of Graphical User Interfaces](<https://www.sota2.com/research/paper/69f2e1e655a3c32af89fc4d0>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2026-05-01T05:00:04.847Z

Authors: Alice Gao, Weixi Tong, Rishab Vempati, Katharina Reinecke, R. Benjamin Shapiro, Tianyi Zhang, Jason Wu

Published: 2026-04-28T18:04:11.000Z

## Abstract

Usability testing with experts and potential users can assess the effectiveness, efficiency, and user satisfaction of graphical user interfaces (GUIs) but doing so remains a costly and time-intensive process. Prior work has used computer use agents (CUAs) and other generative agents that can simulate user interactions and preference, but we show that agents still struggle to provide accurate usability assessments. In this work, we present a novel machine learning method that operationalizes a computational definition of usability to train CUAs to assess GUI usability by i) prioritizing important interaction flows, ii) executing them through human-like interactions, and iii) predicting a learned numerical usability score. We train a computer use agent, uxCUA, with our algorithm on a large-scale dataset of fully interactive user interfaces (UIs) paired with usability labels and human preferences. We show that uxCUA outperforms larger models in accurate usability assessments and produces realistic critiques of both synthetic and real UIs. More broadly, our work aims to build a principled, data-driven foundation for automated usability assessment in HCI.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Usability Assessment on uxWeb (Ground Truths)](<https://www.sota2.com/research/sota/usability-assessment-on-uxweb-ground-truths.md>): Usability Assessment; uxWeb (Ground Truths); AUC: 0.632.

- [Usability Assessment on uxWeb Designer Preferences](<https://www.sota2.com/research/sota/usability-assessment-on-uxweb-designer-preferences.md>): Usability Assessment; uxWeb Designer Preferences; AUC: 60.7.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2604.26020>)

[PDF](<https://arxiv.org/pdf/2604.26020.pdf>)