# Vision-Language System using Open-Source LLMs for Gestures in Medical Interpreter Robots Source: [Vision-Language System using Open-Source LLMs for Gestures in Medical Interpreter Robots]() Collection index: [llms.txt]() Data updated: 2026-03-10T05:00:05.424Z Authors: Thanh-Tung Ngo, Emma Murphy, Robert J. Ross Published: 2026-03-05T23:21:39.000Z ## Abstract Effective communication is vital in healthcare, especially across language barriers, where non-verbal cues and gestures are critical. This paper presents a privacy-preserving vision-language framework for medical interpreter robots that detects specific speech acts (consent and instruction) and generates corresponding robotic gestures. Built on locally deployed open-source models, the system utilizes a Large Language Model (LLM) with few-shot prompting for intent detection. We also introduce a novel dataset of clinical conversations annotated for speech acts and paired with gesture clips. Our identification module achieved 0.90 accuracy, 0.93 weighted precision, and a 0.91 weighted F1-Score. Our approach significantly improves computational efficiency and, in user studies, outperforms the speech-gesture generation baseline in human-likeness while maintaining comparable appropriateness. ## Recorded benchmark results A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage. - [Robot Gesture Generation on Clinical Gesture Dataset User Study](): Robot Gesture Generation; Clinical Gesture Dataset User Study; Human-likeness: 5.78. ## Original sources and implementation [Original paper]() [PDF]()