MERCI: A Multimodal Dataset for Personalised and Emotionally-Aware Dialogues
Althubyani, Mohammed Meng, Zhijin Xie, Shengyuan Cruz, Francisco Razzak, Imran Prasad, Mukesh Sandoval, Eduardo B Kocaballi, Baki
Althubyani, Mohammed
Meng, Zhijin
Xie, Shengyuan
Cruz, Francisco
Razzak, Imran
Prasad, Mukesh
Sandoval, Eduardo B
Kocaballi, Baki
Supervisor
Department
Computational Biology
Embargo End Date
Type
Workshop
Date
License
Language
Collections
Research Projects
Organizational Units
Journal Issue
Abstract
The integration of conversational agents into daily life has become increasingly common. However, sustaining deeply engaging and natural interactions remains challenging due to a lack of multimodal datasets capturing personal and emotional nuances. In this paper, we introduce MERCI (Multimodal dataset for Emotionally-aware peRsonalised Conversational In-teractions), a dataset derived from user-robot dialogues involving thirty participants who completed user profile questionnaires covering ten personal topics (e.g., hobbies, music). A conver-sational system called PERCY then engaged with each partici-pant in open-domain conversations, leveraging GPT-4, real-time facial-expression and sentiment analysis to generate contextu-ally appropriate, empathetic responses. MERCI contains 1860 utterances, equating to about 12.5 hours of aligned audio, three-view video, transcripts with timestamps, emotion labels, and sentiment scores. This dataset serves as a reproducible test-bed for tasks such as emotion-aware response generation, multimodal affect recognition, and personalised policy learning. Baseline performance results have been established using advanced models such as BERT, T5, BART, and GPT-3.5/4/4o-mini across gener-ation, regression, and classification. Evaluations through human and automated methods have demonstrated strong naturalness, relevance, and consistency in responses while indicating areas for enhanced personalisation and empathic depth. We expect that MERCI will enhance the development of emotionally intelligent, user-centric conversational AI applications, potentially ranging from social robotics to mental health support.
Citation
M. Althubyani, Z. Meng, S. Xie, F. Cruz, I. Razzak, M. Prasad, E.B. Sandoval, B. Kocaballi, "MERCI: A Multimodal Dataset for Personalised and Emotionally-Aware Dialogues," 2026, pp. 1-7.
Source
Conference
2025 International Conference on Content-Based Multimedia Indexing (CBMI)
Keywords
46 Information and Computing Sciences, 4608 Human-Centred Computing, 3 Good Health and Well Being
Subjects
Source
2025 International Conference on Content-Based Multimedia Indexing (CBMI)
Publisher
IEEE