Loading...
Chameleon: A Multimodal Learning Framework Robust to Missing Modalities
Muhammad Zaigham Zaheer Karthik Nandakumar Muhammad Haris Khan
Muhammad Zaigham Zaheer
Karthik Nandakumar
Muhammad Haris Khan
Supervisor
Department
Computer Vision
Embargo End Date
Type
Journal article
Date
2025
License
Language
English
Collections
Research Projects
Organizational Units
Journal Issue
Abstract
Multimodal learning has demonstrated remarkable performance improvements over unimodal architectures However, multimodal learning methods often exhibit deteriorated performances if one or more modalities are missing This may be attributed to the commonly used multi-branch design containing modality-specific components, making such approaches reliant on the availability of a complete set of modalities In this work, we propose a robust multimodal learning framework, Chameleon, that adapts a common-space visual learning network to align all input modalities To enable this, we present the unification of input modalities into one format by encoding any non-visual modality into visual representations thus making it robust to missing modalities Extensive experiments are performed on multimodal classification task using four textual-visual (Hateful Memes, UPMC Food-101, MM-IMDb, and Ferramenta) and two audio-visual (avMNIST, VoxCeleb) datasets Chameleon not only achieves superior performance when all modalities are present at train/test time but also demonstrates notable resilience in the case of missing modalities © The Author(s) 2025
Co-author(s)
Liaqat MI, Nawaz S, Zaheer MZ, Saeed MS, Sajjad H, De Schepper T, Nandakumar K, Khan MH, Gallo I, Schedl M
Citation
M. I. Liaqat et al., “Chameleon: A Multimodal Learning Framework Robust to Missing Modalities,” Int J Multimed Inf Retr, vol. 14, no. 2, pp. 1–14, Jun. 2025, doi: 10.1007/S13735-025-00370
Source
International Journal of Multimedia Information Retrieval
Conference
Keywords
Missing modalities, Multimodal learning, Vision and other modalities
Subjects
Source
Publisher
Springer Nature