Ep#92: Human Universal Grasping

How can we learn robot grasping from egocentric human video alone? General-purpose dexterous manipulation learning will require a lot of data, and yet robot data is hard to find at scale. Better leveraging human data, then, will be key to general-purpose dexterous manipulation.

Kevin Wu, Irmak Guzey, and Dandan Shan join us to talk about HUG: Human Unversal Grasping. They collect a dataset of 1M frames (27.8 hours) of egocentric human grasping data, and use it to train a new flow-matching algorithm to predict hand pose. They can then predict human hand poses and retarget them to robot hands, allowing for a massive improvement on a wide variety of zero-shot robot grasping tasks in everyday scenes.

To learn more, watch Episode 92 of RoboPapers with Michael Cho and Chris Paxton today!

Abstract

Humans can grasp objects effortlessly, whereas multi-fingered robots are far from this level of generality. We argue that the most natural source of robot grasping data is from humans, who pick up thousands of objects every day. We present HUG, a flow-matching model that generates diverse human grasps for any user-specified object in a single RGB-D image captured from a stereo camera. Using smart glasses, we first collect 1M-HUGs, an egocentric dataset of human grasps spanning 1M frames (27.8 hrs) and 6,707 object instances across 41 buildings. Next, to model the distribution of natural human grasps, our novel flow-matching model fuses RGB and depth observations to output a grasp parameterized by wrist translation, wrist rotation, and MANO hand pose. Predicted grasps can be retargeted to various robot hands, enabling zero-shot grasping in everyday scenes. To standardize evaluation, we build a new simulated benchmark, HUG-Bench, of 90 unseen objects from five geometric categories and various sizes, with metric-scale 3D meshes. We evaluate HUG in the real world on the 30-object test set of HUG-Bench across multiple stereo cameras, robot embodiments, and household environments. HUG outperforms the state-of-the-art grasping baselines by +23% and +34% on our challenging object set. Code, data, benchmark, checkpoints, and an interactive demo are released on our website: this https URL

Learn More

Project Page: https://grasping.io/

ArXiV: https://arxiv.org/abs/2606.17054

Github: https://github.com/KevinyWu/hug

This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit robopapers.substack.com