# H2R: A Human-to-Robot Data Augmentation for Robot Pre-training from Videos

Source: [H2R: A Human-to-Robot Data Augmentation for Robot Pre-training from Videos](<https://www.sota2.com/research/paper/68fa78e11e209131cea7af95>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2026-03-18T05:00:11.457Z

Authors: Guangrun Li, Yaoxu Lyu, Zhuoyang Liu, Chengkai Hou, Jieyu Zhang, Shanghang Zhang

Published: 2025-05-17T09:08:36.000Z

## Abstract

Large-scale pre-training using egocentric human videos has proven effective for robot learning. However, the models pre-trained on such data can be suboptimal for robot learning due to the significant visual gap between human hands and those of different robots. To remedy this, we propose H2R, a human-to-robot data augmentation pipeline that converts egocentric human videos into robot-centric visual data. H2R estimates human hand pose from videos, retargets the motion to simulated robotic arms, removes human limbs via segmentation and inpainting, and composites rendered robot embodiments into the original frames with camera-aligned geometry. This process explicitly bridges the visual gap between human and robot embodiments during pre-training. We apply H2R to augment large-scale egocentric human video datasets such as Ego4D and SSv2. To verify the effectiveness of the augmentation pipeline, we introduce a CLIP-based image-text similarity metric that quantitatively evaluates the semantic fidelity of robot-rendered frames to the original human actions. We evaluate H2R through comprehensive experiments in both simulation and real-world settings. In simulation, H2R consistently improves downstream success rates across four benchmark suites-Robomimic, RLBench, PushT, and CortexBench-yielding gains of 1.3%-10.2% across different visual encoders and policy learning methods. In real-world experiments, H2R improves performance on UR5 and dual-arm Franka/UR5 manipulation platforms, achieving 3.3%-23.3% success rate gains across gripper-based, dexterous, and bimanual tasks. We further demonstrate the potential of H2R in cross-embodiment generalization and its compatibility with vision-language-action models. These results indicate that H2R improves the generalization ability of robotic policies by mitigating the visual discrepancies between human and robot domains.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Robot Manipulation on Real-world Gripper tasks (UR5 + Robotiq)](<https://www.sota2.com/research/sota/robot-manipulation-on-real-world-gripper-tasks-ur5-plus-robotiq.md>): Robot Manipulation; Real-world Gripper tasks (UR5 + Robotiq); PickCube Success Rate: 65.

- [Robot Manipulation on Real-world Leaphand tasks UR5 + Leaphand](<https://www.sota2.com/research/sota/robot-manipulation-on-real-world-leaphand-tasks-ur5-plus-leaphand.md>): Robot Manipulation; Real-world Leaphand tasks UR5 + Leaphand; Grasp Chicken Success Rate: 55.

- [Robot Manipulation on Real-world Franka tasks Dual-arm](<https://www.sota2.com/research/sota/robot-manipulation-on-real-world-franka-tasks-dual-arm.md>): Robot Manipulation; Real-world Franka tasks Dual-arm; Place Toy Success Rate: 40.

- [Robot Manipulation on Real-world Franka (PlaceToy, SweepRubbish, WeighSauce)](<https://www.sota2.com/research/sota/robot-manipulation-on-real-world-franka-placetoy-sweeprubbish-weighsauce.md>): Robot Manipulation; Real-world Franka; PlaceToy Success Rate: 25.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2505.11920>)

[PDF](<https://arxiv.org/pdf/2505.11920.pdf>)
