MIRA: A Novel Framework for Fusing Modalities in Medical RAG
Wang, Jinhong Ashraf, Tajamul Han, Zongyan Laaksonen, Jorma Anwer, Rao Muhammad
Wang, Jinhong
Ashraf, Tajamul
Han, Zongyan
Laaksonen, Jorma
Anwer, Rao Muhammad
Author
Supervisor
Department
Computer Vision
Embargo End Date
Type
Conference proceeding
Date
2025
License
Language
English
Collections
Research Projects
Organizational Units
Journal Issue
Abstract
Multimodal Large Language Models (MLLM) have significantly advanced AI-assisted medical diagnosis, but often generate factually inconsistent responses that deviate from established medical knowledge. Retrieval-Augmented Generation (RAG) enhances factual accuracy by integrating external sources, but it presents two key challenges. First, insufficient retrieval can miss critical information, whereas excessive retrieval can introduce irrelevant or misleading content, disrupting model output. Second, even when the model initially provides correct answers, over-reliance on retrieved data can lead to factual errors. To address these issues, we introduce Multimodal Intelligent Retrieval and Augmentation (MIRA) framework, designed to optimize factual accuracy in MLLM. MIRA consists of two key components: (1) a calibrated Rethinking and Rearrangement module that dynamically adjusts the number of retrieved contexts to manage factual risk, and (2) A medical RAG framework integrating image embeddings and a medical knowledge base with a query-rewrite module for efficient multimodal reasoning. This enables the model to effectively integrate both its inherent knowledge and external references. Our evaluation of publicly available medical VQA and report generation benchmarks demonstrates that MIRA substantially enhances factual accuracy and overall performance, achieving new state-of-the-art results. Code is released at https://github.com/mbzuai-oryx/MIRA.
Citation
J. Wang, T. Ashraf, Z. Han, J. Laaksonen, and R. M. Anwer, “MIRA: A Novel Framework for Fusing Modalities in Medical RAG,” Proceedings of the 33rd ACM International Conference on Multimedia, pp. 6307–6315, Oct. 2025, doi: 10.1145/3746027.3755760
Source
MSMA '25: Proceedings of the 1st International Workshop on Multi-Sensorial Media and Applications
Conference
The 33rd ACM International Conference on Multimedia
Keywords
Large Language Models, Retrieval Augmented Generation, Medical Reasoning, Visual Question Answering
Subjects
Source
The 33rd ACM International Conference on Multimedia
Publisher
Association for Computing Machinery