# Detours for Navigating Instructional Videos Source: [Detours for Navigating Instructional Videos]() Collection index: [llms.txt]() Data updated: 2026-01-02T05:50:49.754Z Authors: Kumar Ashutosh, Zihui Xue, Tushar Nagarajan, Kristen Grauman Published: 2024-01-03T16:38:56.000Z ## Abstract We introduce the video detours problem for navigating instructional videos. Given a source video and a natural language query asking to alter the how-to video's current path of execution in a certain way, the goal is to find a related ''detour video'' that satisfies the requested alteration. To address this challenge, we propose VidDetours, a novel video-language approach that learns to retrieve the targeted temporal segments from a large repository of how-to's using video-and-text conditioned queries. Furthermore, we devise a language-based pipeline that exploits how-to video narration text to create weakly supervised training data. We demonstrate our idea applied to the domain of how-to cooking videos, where a user can detour from their current recipe to find steps with alternate ingredients, tools, and techniques. Validating on a ground truth annotated dataset of 16K samples, we show our model's significant improvements over best available methods for video retrieval and question answering, with recall rates exceeding the state of the art by 35%. ## Recorded benchmark results A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage. - [Detour video retrieval on Detours (test)](): Detour video retrieval; Detours (test); R@5: 17.6. - [Detour window localization on Detours (test)](): Detour window localization; Detours (test); R@1 (IoU 0.3): 16.7. - [Video demonstration retrieval on COIN CT Woodworking (test)](): Video demonstration retrieval; COIN CT Woodworking (test); Mean Rank: 31. - [Video demonstration retrieval on SaD-MC Woodworking (test)](): Video demonstration retrieval; SaD-MC Woodworking (test); MR: 30. - [Video demonstration retrieval on SaD-MC Gardening (test)](): Video demonstration retrieval; SaD-MC Gardening (test); Mean Rank: 34. - [Video demonstration retrieval on COIN CT (Gardening) (test)](): Video demonstration retrieval; COIN CT (Gardening) (test); MR (Mean Rank): 40. - [Video demonstration retrieval on SaD-MC Cooking (test)](): Video demonstration retrieval; SaD-MC Cooking (test); MR: 124. - [Video demonstration retrieval on COIN CT (Cooking) (test)](): Video demonstration retrieval; COIN CT (Cooking) (test); MR: 37. - [Video demonstration retrieval on SaD-VD Cooking (test)](): Video demonstration retrieval; SaD-VD Cooking (test); MR: 80. - [Video demonstration retrieval on HT-Step Cooking (test)](): Video demonstration retrieval; HT-Step Cooking (test); Mean Rank (MR): 125. ## Original sources and implementation [Original paper]() [PDF]()