Crossing the Human-Robot Embodiment Gap with Sim-to-Real RL using One Human Demonstration

Teaching robots dexterous manipulation skills often requires collecting hundreds of demonstrations using wearables or teleoperation, which is challenging to scale. Videos of human-object interactions are easier to collect and scale, but leveraging them directly for robot learning is difficult due to the lack of explicit action labels and morphological differences between robot and human hands.

We propose Human2Sim2Robot, a novel real-to-sim-to-real framework for training dexterous manipulation policies using only one RGB-D video of a human demonstrating a task. Our method utilizes reinforcement learning (RL) in simulation to cross the human-robot embodiment gap without relying on wearables, teleoperation, or large-scale data collection typically necessary for imitation learning methods. From the demonstration, we extract two task-specific components: (1) the object pose trajectory to define an object-centric, embodiment-agnostic reward function, and (2) the pre-manipulation hand pose to initialize and guide exploration during RL training. We found that these two components are highly effective for learning the desired task, eliminating the need for task-specific reward shaping and tuning. We demonstrate that Human2Sim2Robot significantly outperforms trajectory retargeting and one-shot imitation learning across a wide range of tasks, including grasping, non-prehensile manipulation, and extrinsic manipulation.