Loading...
MSF-Net: Multi-stage fusion network for emotion recognition from multimodal signals in scalable healthcare
Islam, Md. Milon Karray, Fakhri Muhammad, Ghulam
Islam, Md. Milon
Karray, Fakhri
Muhammad, Ghulam
Supervisor
Department
Machine Learning
Embargo End Date
Type
Journal article
Date
2025
License
Language
English
Collections
Research Projects
Organizational Units
Journal Issue
Abstract
Automatic emotion recognition has attracted significant interest in healthcare, thanks to remarkable developments made recently in smart and innovative technologies. A real-time emotion recognition system allows for continuous monitoring, comprehension, and enhancement of the physical entity’s capacities, along with continuing advice for enhancing quality of life and well-being in the context of personalized healthcare. Multimodal emotion recognition presents a significant challenge in terms of efficiently using the diverse modalities present in the data. In this article, we introduce a Multi-Stage Fusion Network (MSF-Net) for emotion recognition capable of extracting multimodal information and achieving significant performances. We propose utilizing the transformer-based structure to extract deep features from facial expressions. We exploited two visual descriptors, local binary pattern and Oriented FAST and Rotated BRIEF, to retrieve the computer vision-based features from the facial videos. A feature-level fusion network integrates the extraction of features from these modules, directing the output into the triplet attention technique. This module employs a three-branch architecture to compute attention weights to capture cross-dimensional interactions efficiently. The temporal dependencies in physiological signals are modeled by a Bi-directional Gated Recurrent Unit (Bi-GRU) in forward and backward directions at each time step. Lastly, the output feature representations from the triplet attention module and the extracted high-level patterns from Bi-GRU are fused and fed into the classification module to recognize emotion. The extensive experimental evaluations revealed that the proposed MSF-Net outperformed the state-of-the-art approaches on two popular datasets, BioVid Emo DB and MGEED. Finally, we tested the proposed MSF-Net in the Internet of Things environment to facilitate real-world scalable smart healthcare application
Citation
Md. M. Islam, F. Karray, and G. Muhammad, “MSF-Net: Multi-stage fusion network for emotion recognition from multimodal signals in scalable healthcare,” Information fusion, vol. 119, pp. 103028-, 2025, doi: 10.1016/j.inffus.2025.103028
Source
Information Fusion
Conference
Keywords
Multimodal emotion recognitionMulti-stage fusionVision transformerBi-directional Gated Recurrent UnitTriplet attentionScalable healthcare
Subjects
Source
Publisher
Elsevier