# Adaptive Vision-Language Model Routing for Computer Use Agents

Source: [Adaptive Vision-Language Model Routing for Computer Use Agents](<https://www.sota2.com/research/paper/69b78e6c0636e5bf4afbdaeb>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2026-03-17T05:00:04.157Z

Authors: Xunzhuo Liu, Bowei He, Xue Liu, Andy Luo, Haichen Zhang, Huamin Chen

Published: 2026-03-13T09:21:25.000Z

## Abstract

Computer Use Agents (CUAs) translate natural-language instructions into Graphical User Interface (GUI) actions such as clicks, keystrokes, and scrolls by relying on a Vision-Language Model (VLM) to interpret screenshots and predict grounded tool calls. However, grounding accuracy varies dramatically across VLMs, while current CUA systems typically route every action to a single fixed model regardless of difficulty. We propose \\textbf{Adaptive VLM Routing} (AVR), a framework that inserts a lightweight semantic routing layer between the CUA orchestrator and a pool of VLMs. For each tool call, AVR estimates action difficulty from multimodal embeddings, probes a small VLM to measure confidence, and routes the action to the cheapest model whose predicted accuracy satisfies a target reliability threshold. For \\textit{warm} agents with memory of prior UI interactions, retrieved context further narrows the capability gap between small and large models, allowing many actions to be handled without escalation. We formalize routing as a cost--accuracy trade-off, derive a threshold-based policy for model selection, and evaluate AVR using ScreenSpot-Pro grounding data together with the OpenClaw agent routing benchmark. Across these settings, AVR projects inference cost reductions of up to 78\\% while staying within 2 percentage points of an all-large-model baseline. When combined with the Visual Confused Deputy guardrail, AVR also escalates high-risk actions directly to the strongest available model, unifying efficiency and safety within a single routing framework. Materials are also provided Model, benchmark, and code: https://github.com/vllm-project/semantic-router.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [CUA grounding on CUA grounding analytical projection OpenClaw](<https://www.sota2.com/research/sota/cua-grounding-on-cua-grounding-analytical-projection-openclaw.md>): CUA grounding; CUA grounding analytical projection OpenClaw; Esc. Alpha: 0.35.

- [GUI Grounding on ScreenSpot Pro](<https://www.sota2.com/research/sota/gui-grounding-on-screenspot-pro.md>): GUI Grounding; ScreenSpot Pro; Accuracy: not recorded.

- [VLM Grounding on OSWorld](<https://www.sota2.com/research/sota/vlm-grounding-on-osworld.md>): VLM Grounding; OSWorld; Accuracy: not recorded.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2603.12823>)

[PDF](<https://arxiv.org/pdf/2603.12823.pdf>)