DexMimicGen: Automated Data Generation for Bimanual Dexterous Manipulation via Imitation Learning

Zhenyu Jiang*1,2, Yuqi Xie*1,2, Kevin Lin*1,2, Zhenjia Xu1, Weikang Wan3,
Ajay Mandlekar†1, Linxi "Jim" Fan†1, Yuke Zhu†1,2

*Equal Contributions, †Project Leads

1NVIDIA Research 2UT Austin 3UC San Diego
Paper arXiv Summary


DexMimicGen automatically generates large datasets for bimanual dexterous manipulation from a few human demonstrations

DexMimicGen produces large-scale bimanual dexterous manipulation datasets with minimal human effort

We used DexMimicGen to autonomously generate over 20,000 demonstrations for bimanual dexterous robots from just 60 source human demonstrations across 9 tasks, multiple simulators, and the real-world.

MimicGen datasets can produce performant policies across diverse tasks with simple Behavioral Cloning

Can Sorting

Success Rate: 97.3 ± 0.9%

Coffee

Success Rate: 77.3 ± 0.9%

Pouring

Success Rate: 79.3 ± 0.9%

Tray Lift

Success Rate: 88.7 ± 0.9%

Box Cleanup

Success Rate: 92.0 ± 4.3%

Drawer Cleanup

Success Rate: 76.0 ± 0.0%

Threading

Success Rate: 69.3 ± 1.9%

Transport

Success Rate: 83.3 ± 0.9%

Piece Assembly

Success Rate: 80.7 ± 0.9%

Policy performance improves with larger DexMimicGen datasets

DexMimicGen leverages a Real2Sim2Real pipeline to train performant policies for the real world

DexMimicGen can generate diverse datasets from few human demonstrations for new reset distributions

We showcase datasets generated by DexMimicGen across different reset distributions for three tasks in Robosuite below.

Pouring

Reset Distribution D0

Reset Distribution D1

Reset Distribution D2

Box Cleanup

Reset Distribution D0

Reset Distribution D1

Reset Distribution D2

Three Piece Assembly

Reset Distribution D0

Reset Distribution D1

Reset Distribution D2

Drawer Cleanup

Reset Distribution D0

Reset Distribution D1

DexMimicGen generates datasets for different simulation frameworks

The following generated datasets are from the BiGym simulation framework and feature mobile bimanual manipulation, while the other visualized datasets come from the Robosuite.

Dishwasher Load Plates

Flip Cup

Put Cups

DexMimicGen Pipeline Overview

BibTeX

@inproceedings{jiang2024dexmimicen,
      title     = {DexMimicGen: Automated Data Generation for Bimanual Dexterous Manipulation via Imitation Learning},
      author    = {Jiang, Zhenyu and Xie, Yuqi and Lin, Kevin and Xu, Zhenjia and Wan, Weikang and Mandlekar, Ajay and Fan, Linxi and Zhu, Yuke},
      journal   = {arXiv preprint arXiv:2410.24185},
      year      = {2024}
}

Contact

If you have any questions, please feel free to contact Zhenyu Jiang, Ajay Mandelkar, Jim Fan and Yuke Zhu.