# What to Say and When to Say it: Live Fitness Coaching as a Testbed for Situated Interaction Source: [What to Say and When to Say it: Live Fitness Coaching as a Testbed for Situated Interaction]() Collection index: [llms.txt]() Data updated: 2026-04-17T05:00:04.440Z Authors: Sunny Panchal, Apratim Bhattacharyya, Guillaume Berger, Antoine Mercier, Cornelius Bohm, Florian Dietrichkeit, Reza Pourreza, Xuanlin Li, Pulkit Madan, Mingu Lee, Mark Todorovich, Ingo Bax, Roland Memisevic Published: 2024-07-11T00:10:45.000Z ## Abstract Vision-language models have shown impressive progress in recent years. However, existing models are largely limited to turn-based interactions, where each turn must be stepped (i.e., prompted) by the user. Open-ended, asynchronous interactions, where an AI model may proactively deliver timely responses or feedback based on the unfolding situation in real-time, are an open challenge. In this work, we present the QEVD benchmark and dataset, which explores human-AI interaction in the challenging, yet controlled, real-world domain of fitness coaching -- a task which intrinsically requires monitoring live user activity and providing immediate feedback. The benchmark requires vision-language models to recognize complex human actions, identify possible mistakes, and provide appropriate feedback in real-time. Our experiments reveal the limitations of existing state-of-the-art vision-language models for such asynchronous situated interactions. Motivated by this, we propose a simple end-to-end streaming baseline that can respond asynchronously to human actions with appropriate feedback at the appropriate time. ## Recorded benchmark results A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage. - [Interactive Video Feedback Generation on QEVD-FIT-COACH](): Interactive Video Feedback Generation; QEVD-FIT-COACH; METEOR: 0.127. - [Video-to-Feedback Generation on QEVD-fit-coach (original annotations)](): Video-to-Feedback Generation; QEVD-fit-coach (original annotations); METEOR: 12.7. - [Streaming Biomechanical Feedback Generation on QEVD bio-fit-coach](): Streaming Biomechanical Feedback Generation; QEVD bio-fit-coach; METEOR Score: 0.086. - [Video Feedback Generation on QEVD-FIT-COACH](): Video Feedback Generation; QEVD-FIT-COACH; METEOR: not recorded. ## Original sources and implementation [Original paper]() [PDF]()