# V2CNet: A Deep Learning Framework to Translate Videos to Commands for Robotic Manipulation

Source: [V2CNet: A Deep Learning Framework to Translate Videos to Commands for Robotic Manipulation](<https://www.sota2.com/research/paper/68fa65841e209131cea0ab8e>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2025-10-25T21:28:43.855Z

Authors: Anh Nguyen, Thanh-Toan Do, Ian Reid, Darwin G. Caldwell, Nikos G. Tsagarakis

Published: 2019-03-23T04:02:51.000Z

## Abstract

We propose V2CNet, a new deep learning framework to automatically translate the demonstration videos to commands that can be directly used in robotic applications. Our V2CNet has two branches and aims at understanding the demonstration video in a fine-grained manner. The first branch has the encoder-decoder architecture to encode the visual features and sequentially generate the output words as a command, while the second branch uses a Temporal Convolutional Network (TCN) to learn the fine-grained actions. By jointly training both branches, the network is able to model the sequential information of the command, while effectively encodes the fine-grained actions. The experimental results on our new large-scale dataset show that V2CNet outperforms recent state-of-the-art methods by a substantial margin, while its output can be applied in real robotic applications. The source code and trained models will be made available.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Video-to-command generation on standard object sets](<https://www.sota2.com/research/sota/video-to-command-generation-on-standard-object-sets.md>): Video-to-command generation; standard object sets; BLEU-1: 0.357.

- [Video-to-command generation on Modified Something-Something Standard Object Sets V2 (test)](<https://www.sota2.com/research/sota/video-to-command-generation-on-modified-something-something-standard-object-sets-v2-test.md>): Video-to-command generation; Modified Something-Something Standard Object Sets V2 (test); BLEU-1: 0.357.

- [Video-to-command generation on Modified Something-Something V2 (Novel Object Sets)](<https://www.sota2.com/research/sota/video-to-command-generation-on-modified-something-something-v2-novel-object-sets.md>): Video-to-command generation; Modified Something-Something V2 (Novel Object Sets); BLEU-1: 0.259.

- [Video-to-command generation on Novel object sets Zero-shot (test)](<https://www.sota2.com/research/sota/video-to-command-generation-on-novel-object-sets-zero-shot-test.md>): Video-to-command generation; Novel object sets Zero-shot (test); BLEU-2: 11.9.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/1903.10869>)

[PDF](<https://arxiv.org/pdf/1903.10869.pdf>)
