# HOI-Ref: Hand-Object Interaction Referral in Egocentric Vision Source: [HOI-Ref: Hand-Object Interaction Referral in Egocentric Vision]() Collection index: [llms.txt]() Data updated: 2025-10-26T02:37:39.153Z Authors: Siddhant Bansal, Michael Wray, Dima Damen Published: 2024-04-15T16:59:00.000Z ## Abstract Large Vision Language Models (VLMs) are now the de facto state-of-the-art for a number of tasks including visual question answering, recognising objects, and spatial referral. In this work, we propose the HOI-Ref task for egocentric images that aims to understand interactions between hands and objects using VLMs. To enable HOI-Ref, we curate the HOI-QA dataset that consists of 3.9M question-answer pairs for training and evaluating VLMs. HOI-QA includes questions relating to locating hands, objects, and critically their interactions (e.g. referring to the object being manipulated by the hand). We train the first VLM for HOI-Ref on this dataset and call it VLM4HOI. Our results demonstrate that VLMs trained for referral on third person images fail to recognise and refer hands and objects in egocentric images. When fine-tuned on our egocentric HOI-QA dataset, performance improves by 27.9% for referring hands and objects, and by 26.7% for referring interactions. ## Recorded benchmark results A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage. - [Temporal contact detection on FEEL (val)](): Temporal contact detection; FEEL (val); BC: 59.6. - [Temporal contact detection on VISOR (val)](): Temporal contact detection; VISOR (val); BC: 55.4. - [Temporal contact detection on HOI4D (val)](): Temporal contact detection; HOI4D (val); BC: 52.1. ## Original sources and implementation [Original paper]() [PDF]()