Finger Gesture Tracking From Arbitrary Hand Position Using Smart Skin Technology and Deep Learning
Marew, Rufael
Marew, Rufael
Author
Supervisor
Department
Machine Learning
Embargo End Date
2025-05-21
Type
Thesis
Date
2024
License
Language
English
Collections
Research Projects
Organizational Units
Journal Issue
Abstract
The exploration of new approaches to achieve finger gesture recognition has been a subject of extensive study, with particular interest in leveraging Electromyography (EMG). The allure of EMG lies in its potential utility in applications where visual imaging is not practical. Until recently, EMG data required a significant reliance on hand position, often presenting intricate challenges in distinguishing between overlapping patterns associated with various positions. This study proposes a novel solution by employing a transformer model and motion sensor-based training approach to achieve unprecedented accuracy and utility. We tested our approach on eight subjects performing 10 distinct finger gestures and arbitrary hand positions. We designed a dedicated dual-branch transformer architecture with the following features: (1) it utilizes spatio-temporal patches of the input to decode the intricate relationship between spatial and temporal features present in EMG (2) it maps the temporal and spectral representations of the signal to get a holistic feature map. (3) it aligns the feature maps of time and frequency by introducing a time-frequency loss term to the training objective. With this, the model was able to make robust predictions with varying hand positions dramatically enhancing the application of EMG in finger gesture recognition over previously reported approaches.
Citation
R. Marew, "Finger Gesture Tracking From Arbitrary Hand Position Using Smart Skin Technology and Deep Learning", MS. Thesis, Machine Learning, MBZUAI, Abu Dhabi, UAE, 2024