# iTeach: In the Wild Interactive Teaching for Failure-Driven Adaptation of Robot Perception

Source: [iTeach: In the Wild Interactive Teaching for Failure-Driven Adaptation of Robot Perception](<https://www.sota2.com/research/paper/68fa78741e209131cea78697>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2026-04-16T05:00:04.683Z

Authors: Jishnu Jaykumar P, Cole Salvato, Vinaya Bomnale, Jikai Wang, Yu Xiang

Published: 2024-10-01T18:30:59.000Z

## Abstract

Robotic perception models often fail when deployed in real-world environments due to out-of-distribution conditions such as clutter, occlusion, and novel object instances. Existing approaches address this gap through offline data collection and retraining, which are slow and do not resolve deployment-time failures. We propose iTeach, a failure-driven interactive teaching framework for adapting robot perception in the wild. A co-located human observes model predictions during deployment, identifies failure cases, and performs short human-object interaction (HumanPlay) to expose informative object configurations while recording RGB-D sequences. To minimize annotation effort, iTeach employs a Few-Shot Semi- Supervised (FS3) labeling strategy, where only the final frame of a short interaction sequence is annotated using hands-free eye-gaze and voice commands, and labels are propagated across the video to produce dense supervision. The collected failure-driven samples are used for iterative fine-tuning, enabling progressive deployment-time adaptation of the perception model. We evaluate iTeach on unseen object instance segmentation (UOIS) starting from a pretrained MSMFormer model. Using a small number of failure-driven samples, our method significantly improves segmentation performance across diverse real-world scenes. These improvements directly translate to higher grasping and pick-and-place success on the SceneReplica benchmark and real robotic experiments. Our results demonstrate that failure-driven, co-located interactive teaching enables efficient in-the-wild adaptation of robot perception and improves downstream manipulation performance. Project page at https://irvlutd.github.io/iTeach

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Robot Grasping on SceneReplica](<https://www.sota2.com/research/sota/robot-grasping-on-scenereplica.md>): Robot Grasping; SceneReplica; Grasping Success Rate: 74.

- [Robot Pick-and-Place on SceneReplica](<https://www.sota2.com/research/sota/robot-pick-and-place-on-scenereplica.md>): Robot Pick-and-Place; SceneReplica; Pick-and-Place Success: 72.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2410.09072>)

[PDF](<https://arxiv.org/pdf/2410.09072.pdf>)