Multimodal Sentiment Analysis: Recognizing Sentiment in Memes
Vankov, Georgi Dimitrov, Dimitar Koychev, Ivan Nakov, Preslav
Vankov, Georgi
Dimitrov, Dimitar
Koychev, Ivan
Nakov, Preslav
Supervisor
Department
Natural Language Processing
Embargo End Date
Type
Conference proceeding
Date
2025
License
Language
English
Collections
Research Projects
Organizational Units
Journal Issue
Abstract
The usage of memes and other visual material coupled with text on social media has been on the rise recently. Recognizing that visual signals are consumed quickly and can trigger emotional responses. It has become essential to discern the sentiment of such content, as it could significantly influence social media users. The paper focuses on the sentiment of memes on popular social networking platforms such as Instagram, Reddit, Facebook, and Tumblr. Our goal is to understand how these memes affect people in a positive, negative, or neutral way. We create a balanced dataset of 5,592 memes using distant supervision, i.e., automatically assigning sentiment labels based on different social media attributes, e.g., hashtags. We verify the accuracy of these labels by manually checking a random subset of the data. We conduct unimodal and multimodal experiments to explore how different cues contribute to identifying sentiment. Our results show that multimodal approaches, combining images and text, effectively identify the emotions in memes. We further experiment with novel closed and open-source LLMs, and we show that they outperform traditional multimodal approaches. The dataset is released publicly.
Citation
G. Vankov, D. Dimitrov, I. Koychev, and P. Nakov, “Multimodal Sentiment Analysis: Recognizing Sentiment in Memes,” Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) , vol. 15462, pp. 1–11, Jan. 2025, doi: 10.1007/978-3-031-81542-3_1.
Source
Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Conference
International Conference on Artificial Intelligence: Methodology, Systems, and Applications (AIMSA)
Keywords
Emotion recognition, Meme dataset, Multimodal Sentiment Analysis
Subjects
Source
International Conference on Artificial Intelligence: Methodology, Systems, and Applications (AIMSA)
Publisher
Springer Nature