MOTOR: Multimodal Optimal Transport via Grounded Retrieval in Medical Visual Question Answering
Shaaban, Mai A. Saleem, Tausifa Jan Papineni, Vijay Ram Kumar Yaqub, Mohammad
Shaaban, Mai A.
Saleem, Tausifa Jan
Papineni, Vijay Ram Kumar
Yaqub, Mohammad
Supervisor
Department
Computer Vision
Embargo End Date
Type
Conference proceeding
Date
2025
License
Language
English
Collections
Research Projects
Organizational Units
Journal Issue
Abstract
Medical visual question answering (MedVQA) plays a vital role in clinical decision-making by providing contextually rich answers to image-based queries. Although vision-language models (VLMs) are widely used for this task, they often generate factually incorrect answers. Retrieval-augmented generation addresses this challenge by providing information from external sources, but risks retrieving irrelevant context, which can degrade the reasoning capabilities of VLMs. Re-ranking retrievals, as introduced in existing approaches, enhances retrieval relevance by focusing on query-text alignment. However, these approaches neglect the visual or multimodal context, which is particularly crucial for medical diagnosis. We propose MOTOR, a novel multimodal retrieval and re-ranking approach that leverages grounded captions and optimal transport. It captures the underlying relationships between the query and the retrieved context based on textual and visual information. Consequently, our approach identifies more clinically relevant contexts to augment the VLM input. Empirical analysis and human expert evaluation demonstrate that MOTOR achieves higher accuracy on MedVQA datasets, outperforming state-of-the-art methods by an average of 6.45%. Code is available at https://github.com/BioMedIA-MBZUAI/MOTOR.
Citation
M. A. Shaaban, T. J. Saleem, V. Ram, K. Papineni, and M. Yaqub, “MOTOR: Multimodal Optimal Transport via Grounded Retrieval in Medical Visual Question Answering,” pp. 459–469, 2026, doi: 10.1007/978-3-032-04978-0_44
Source
Medical Image Computing and Computer Assisted Intervention
Conference
28 MICCAI: International Conference on Medical Image Computing and Computer-Assisted Intervention
Keywords
Medical Visual Question Answering, Retrieval-Augmented Generation, Multimodal Reranker, Optimal Transport, Grounded Caption
Subjects
Source
28 MICCAI: International Conference on Medical Image Computing and Computer-Assisted Intervention
Publisher
Springer Nature