# Translating Videos to Commands for Robotic Manipulation with Deep Recurrent Neural Networks

Source: [Translating Videos to Commands for Robotic Manipulation with Deep Recurrent Neural Networks](<https://www.sota2.com/research/paper/68fa63b91e209131cea00616>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2025-10-28T13:56:34.464Z

Authors: Anh Nguyen, Dimitrios Kanoulas, Luca Muratore, Darwin G. Caldwell, Nikos G. Tsagarakis

Published: 2017-10-01T04:19:52.000Z

## Abstract

We present a new method to translate videos to commands for robotic manipulation using Deep Recurrent Neural Networks (RNN). Our framework first extracts deep features from the input video frames with a deep Convolutional Neural Networks (CNN). Two RNN layers with an encoder-decoder architecture are then used to encode the visual features and sequentially generate the output words as the command. We demonstrate that the translation accuracy can be improved by allowing a smooth transaction between two RNN layers and using the state-of-the-art feature extractor. The experimental results on our new challenging dataset show that our approach outperforms recent methods by a fair margin. Furthermore, we combine the proposed translation module with the vision and planning system to let a robot perform various manipulation tasks. Finally, we demonstrate the effectiveness of our framework on a full-size humanoid robot WALK-MAN.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Video-to-command generation on Novel object sets Zero-shot (test)](<https://www.sota2.com/research/sota/video-to-command-generation-on-novel-object-sets-zero-shot-test.md>): Video-to-command generation; Novel object sets Zero-shot (test); BLEU-2: 15.

- [Video-to-command generation on Modified Something-Something V2 (Novel Object Sets)](<https://www.sota2.com/research/sota/video-to-command-generation-on-modified-something-something-v2-novel-object-sets.md>): Video-to-command generation; Modified Something-Something V2 (Novel Object Sets); BLEU-1: 0.305.

- [Video-to-command generation on Modified Something-Something Standard Object Sets V2 (test)](<https://www.sota2.com/research/sota/video-to-command-generation-on-modified-something-something-standard-object-sets-v2-test.md>): Video-to-command generation; Modified Something-Something Standard Object Sets V2 (test); BLEU-1: 0.312.

- [Video-to-command generation on standard object sets](<https://www.sota2.com/research/sota/video-to-command-generation-on-standard-object-sets.md>): Video-to-command generation; standard object sets; BLEU-1: 0.323.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/1710.00290>)

[PDF](<https://arxiv.org/pdf/1710.00290.pdf>)
