ViViDex: Learning Vision-Based Dexterous Manipulation from Human Videos
Chen, Zerui Chen, Shizhe Arlaud, Etienne Laptev, Ivan Schmid, Cordelia
Chen, Zerui
Chen, Shizhe
Arlaud, Etienne
Laptev, Ivan
Schmid, Cordelia
Supervisor
Department
Computer Vision
Embargo End Date
Type
Conference proceeding
Date
2025
License
Language
English
Collections
Research Projects
Organizational Units
Journal Issue
Abstract
In this work, we aim to learn a unified vision-based policy for multi-fingered robot hands to manipulate a variety of objects in diverse poses. Though prior work has shown benefits of using human videos for policy learning, performance gains have been limited by the noise in estimated trajectories. Moreover, reliance on privileged object information such as ground-truth object states further limits the applicability in realistic scenarios. To address these limitations, we propose a new framework ViViDex to improve vision-based policy learning from human videos. It first uses reinforcement learning with trajectory guided rewards to train state-based policies for each video, obtaining both visually natural and physically plausible trajectories from the video. We then rollout successful episodes from state-based policies and train a unified visual policy without using any privileged information. We propose coordinate transformation to further enhance the visual point cloud representation, and compare behavior cloning and diffusion policy for the visual policy training. Experiments both in simulation and on the real robot demonstrate that ViViDex outperforms state-of-theart approaches on three dexterous manipulation tasks. Project website: zerchen.github.io/projects/vividex.html.
Citation
Z. Chen, S. Chen, E. Arlaud, I. Laptev and C. Schmid, "ViViDex: Learning Vision-Based Dexterous Manipulation from Human Videos," 2025 IEEE International Conference on Robotics and Automation (ICRA), Atlanta, GA, USA, 2025, pp. 3336-3343, doi: 10.1109/ICRA55743.2025.11127358.
Source
International Conference on Robotics and Automation (ICRA)
Conference
2025 IEEE International Conference on Robotics and Automation (ICRA)
Keywords
Point Cloud Compression, Training, Visualization, Three-Dimensional Displays, Robot Kinematics, Noise, Reinforcement Learning, Performance Gain, Trajectory, Videos
Subjects
Source
2025 IEEE International Conference on Robotics and Automation (ICRA)
Publisher
IEEE