ARCTIC: A Dataset for Dexterous Bimanual Hand-Object Manipulation
About
Humans intuitively understand that inanimate objects do not move by themselves, but that state changes are typically caused by human manipulation (e.g., the opening of a book). This is not yet the case for machines. In part this is because there exist no datasets with ground-truth 3D annotations for the study of physically consistent and synchronised motion of hands and articulated objects. To this end, we introduce ARCTIC -- a dataset of two hands that dexterously manipulate objects, containing 2.1M video frames paired with accurate 3D hand and object meshes and detailed, dynamic contact information. It contains bi-manual articulation of objects such as scissors or laptops, where hand poses and object states evolve jointly in time. We propose two novel articulated hand-object interaction tasks: (1) Consistent motion reconstruction: Given a monocular video, the goal is to reconstruct two hands and articulated objects in 3D, so that their motions are spatio-temporally consistent. (2) Interaction field estimation: Dense relative hand-object distances must be estimated from images. We introduce two baselines ArcticNet and InterField, respectively and evaluate them qualitatively and quantitatively on ARCTIC. Our code and data are available at https://arctic.is.tue.mpg.de.
Key insights
What problem does ARCTIC address?
ARCTIC enables study of two hands dexterously manipulating articulated objects like laptops or scissors, where hand poses and object states evolve jointly. It provides 2.1M RGB frames with 3D meshes and contact data, captured via 54-camera Vicon MoCap.
How does ARCTIC differ from prior datasets?
Unlike prior datasets limited to single-hand grasping of rigid objects, ARCTIC uniquely combines both hands, full body, articulated objects, and dexterous manipulation across 9 views. T-SNE shows significantly broader hand pose diversity than existing datasets.
What are the practical uses of these findings?
ARCTIC provides benchmark baselines for consistent motion reconstruction and interaction field estimation. Pre-training on ARCTIC then fine-tuning on HO3D improves hand MPJPE by 9.2% and object vertex error by 7.1%, demonstrating transferable benefits.
What limitations do the authors acknowledge?
Baselines assume known object models, not unknown articulated shapes. Some objects are toys lacking real-world visual complexity. Objects have only 1 degree of freedom. SMPL-X does not capture skin deformation during contact.