# Referring to Screen Texts with Voice Assistants Source: [Referring to Screen Texts with Voice Assistants]() Collection index: [llms.txt]() Data updated: 2026-01-02T02:58:12.673Z Authors: Shruti Bhargava, Anand Dhoot, Ing-Marie Jonsson, Hoang Long Nguyen, Alkesh Patel, Hong Yu, Vincent Renkens Published: 2023-06-10T22:43:16.000Z ## Abstract Voice assistants help users make phone calls, send messages, create events, navigate, and do a lot more. However, assistants have limited capacity to understand their users' context. In this work, we aim to take a step in this direction. Our work dives into a new experience for users to refer to phone numbers, addresses, email addresses, URLs, and dates on their phone screens. Our focus lies in reference understanding, which becomes particularly interesting when multiple similar texts are present on screen, similar to visual grounding. We collect a dataset and propose a lightweight general-purpose model for this novel experience. Due to the high cost of consuming pixels directly, our system is designed to rely on the extracted text from the UI. Our model is modular, thus offering flexibility, improved interpretability, and efficient runtime memory utilization. ## Recorded benchmark results A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage. - [Screen Reference Resolution on ScreenRef Descriptive](): Screen Reference Resolution; ScreenRef Descriptive; Top-1 Error: 14.2. - [Screen Reference Resolution on ScreenRef Category-level](): Screen Reference Resolution; ScreenRef Category-level; Top-1 Error: 0.011. ## Original sources and implementation [Original paper]() [PDF]()