Loading...
MSAmba: Exploring Multimodal Sentiment Analysis with State Space Models
He, Xilin Liang, Haijian Peng, Boyi Xie, Weicheng Khan, Muhammad Haris Song, Siyang Yu, Zitong
He, Xilin
Liang, Haijian
Peng, Boyi
Xie, Weicheng
Khan, Muhammad Haris
Song, Siyang
Yu, Zitong
Supervisor
Department
Computer Vision
Embargo End Date
Type
Conference proceeding
Date
2025
License
Language
English
Collections
Research Projects
Organizational Units
Journal Issue
Abstract
Multimodal sentiment analysis, which learns a model to process multiple modalities simultaneously and predict a sentiment value, is an important area of affective computing. Modeling sequential intra-modal information and enhancing cross-modal interactions are crucial to multimodal sentiment analysis. In this paper, we propose MSAmba, a novel hybrid Mamba-based architecture for multimodal sentiment analysis, consisting of two core blocks: Intra-Modal Sequential Mamba (ISM) block and Cross-Modal Hybrid Mamba (CHM) block, to comprehensively address the above-mentioned challenges with hybrid state space models. Firstly, the ISM block models the sequential information within each modality in a bi-directional manner with the assistance of global information. Subsequently, the CHM blocks explicitly model centralized cross-modal interaction with a hybrid combination of Mamba and attention mechanism to facilitate information fusion across modalities. Finally, joint learning of the intra-modal tokens and cross-modal tokens is utilized to predict the sentiment values. This paper serves as one of the pioneering works to unravel the outstanding performances and great research potential of Mamba-based methods in the task of multimodal sentiment analysis. Experiments on CMU-MOSI, CMU-MOSEI and CH-SIMS demonstrate the superior performance of the proposed MSAmba over prior Transformer-based and CNN-based methods.
Citation
X. He et al., “MSAmba: Exploring Multimodal Sentiment Analysis with State Space Models,” Proceedings of the AAAI Conference on Artificial Intelligence, vol. 39, no. 2, pp. 1309–1317, Apr. 2025, doi: 10.1609/AAAI.V39I2.32120.
Source
Proceedings of the AAAI Conference on Artificial Intelligence
Conference
39th Annual AAAI Conference on Artificial Intelligence, AAAI 2025
Keywords
Information fusion, Space research, Affective Computing, Cross-modal, Cross-modal interaction, Hybrid state-space models, Learn+, Multi-modal, Multiple modalities, Performance, Sentiment analysis, State-space models, Modal analysis
Subjects
Source
39th Annual AAAI Conference on Artificial Intelligence, AAAI 2025
Publisher
Association for the Advancement of Artificial Intelligence