A Complementary Approach for Robust and Safety-Oriented Visual Tracking via Near-Infrared and RGB-D Cameras for Safe Physical Human-Robot Interaction
Hamad, Mazin Kangwagye, Samuel Le Mesle, Valentin Mosberger, Rafael Lilienthal, Achim J Haddadin, Sami
Hamad, Mazin
Kangwagye, Samuel
Le Mesle, Valentin
Mosberger, Rafael
Lilienthal, Achim J
Haddadin, Sami
Supervisor
Department
Robotics
Embargo End Date
Type
Conference proceeding
Date
License
Language
Collections
Research Projects
Organizational Units
Journal Issue
Abstract
Safe physical human-robot interaction (pHRI) in industrial settings requires robust and accurate tracking of key points on the human co-worker’s body and the robot structure. However, vision-based single-camera tracking solutions often face many challenges, such as limited field of view (FoV), detection range, occlusions, and inconsistent detection. This paper proposes a complementary multi-sensor tracking scheme that integrates RGB-D and near-infrared (NIR) cameras to improve human motion tracking accuracy while ensuring compliance with ISO/TS 15066 safety requirements. For the first time in pHRI, we deploy an infrared-based tracking system, originally designed for driver assistance and accident prevention, to complement RGB-D cameras, which provide detailed pose estimation at near range but suffer from a narrow FoV. A safety-oriented complementary approach is developed to fuse human tracking data from both systems into robot control, integrating a well-established safety paradigm based on the Safe Motion Unit (SMU) framework. The proposed system is experimentally validated in real-world collaborative robotic workspaces across various pHRI scenarios. Results demonstrate its effectiveness in respecting human safety constraints, even under challenging operating conditions, without unnecessary performance restrictions. The complementary vision-based approach improves tracking accuracy, expands FoV, and enhances reliability, making it a promising solution for certifiable, human-aware collaborative robotics in various industrial settings. The video documentation can be seen at https://youtu.be/xWksc_vhuew.
Citation
M. Hamad, S. Kangwagye, V. Le Mesle, R. Mosberger, A.J. Lilienthal, S. Haddadin, "A Complementary Approach for Robust and Safety-Oriented Visual Tracking via Near-Infrared and RGB-D Cameras for Safe Physical Human-Robot Interaction," 2026, pp. 584-591.
Source
Conference
2026 IEEE/SICE International Symposium on System Integration (SII)
Keywords
40 Engineering, 4007 Control Engineering, Mechatronics and Robotics, 46 Information and Computing Sciences, 4605 Data Management and Data Science, 4608 Human-Centred Computing
Subjects
Source
2026 IEEE/SICE International Symposium on System Integration (SII)
Publisher
IEEE