Multi-modal vision-based deformable perception for in-finger manipulation with soft active surfaces
Li, Sen Song, Chaoyang Wan, Fang
Li, Sen
Song, Chaoyang
Wan, Fang
Author
Supervisor
Department
Others
Embargo End Date
Type
Journal article
Date
License
http://creativecommons.org/licenses/by/4.0/
Language
English
Collections
Research Projects
Organizational Units
Journal Issue
Abstract
Enabling robots to manipulate objects dexterously within their grasp often requires complex multi-fingered hands with dense, multi-modal sensor arrays. This complexity can limit practicality and robustness, especially in confined spaces. This study challenges that paradigm by introducing a robust multi-modal perception method that derives 6D forces and poses from the large-scale deformation of a soft finger, using only a single internal camera. We propose ActiveSPN2, a robot gripper with a soft active surface designed for in-finger manipulation in constrained environments, featuring multi-modal perception through a vision-based deformable perception architecture. By tracking the real-time deformation of the internal Soft Polyhedral Network, we achieve real-time, multi-modal vision-based perception that provides 6D forces and object pose estimation during in-finger manipulation. Quantitatively, the proposed force perception model achieves an average accuracy of approximately 0.286 N on the test set, indicating reliable generalization under unseen manipulation conditions. A series of experiments were conducted on the ActiveSPN2 gripper using the proposed vision-based deformable perception approach, demonstrating the effectiveness and robustness of the strategy.
Citation
S. Li, C. Song, F. Wan, "Multi-modal vision-based deformable perception for in-finger manipulation with soft active surfaces," Biomimetic Intelligence and Robotics, pp. 100326-100326, 2026, https://doi.org/10.1016/j.birob.2026.100326.
Source
Biomimetic Intelligence and Robotics
Conference
Keywords
46 Information and Computing Sciences, 4608 Human-Centred Computing
Subjects
Source
Publisher
Elsevier