MedPromptX: Grounded Multimodal Prompting for Chest X-Ray Diagnosis
Shaaban, Mai A. Khan, Adnan Yaqub, Mohammad
Shaaban, Mai A.
Khan, Adnan
Yaqub, Mohammad
Supervisor
Department
Computer Vision
Embargo End Date
Type
Conference proceeding
Date
2025
License
Language
English
Collections
Research Projects
Organizational Units
Journal Issue
Abstract
Chest X-ray images are commonly used for predicting acute and chronic cardiopulmonary conditions, but efforts to integrate them with structured clinical data face challenges due to incomplete electronic health records (EHR). This paper introduces MedPromptX, the first clinical decision support system that integrates multimodal large language models (MLLMs), few-shot prompting (FP) and visual grounding (VG) to combine imagery with EHR data for chest X-ray diagnosis. A pre-trained MLLM is utilized to complement the missing EHR information, providing a comprehensive understanding of patients’ medical history. Additionally, FP reduces the necessity for extensive training of MLLMs while effectively tackling the issue of hallucination. Nevertheless, the process of determining the optimal number of few-shot examples and selecting high-quality candidates can be burdensome, yet it profoundly influences model performance. Hence, we propose a new technique that dynamically refines few-shot data for real-time adjustment to new patient scenarios. Moreover, VG narrows the search area in X-ray images, thereby enhancing the identification of abnormalities. We also release MedPromptX-VQA, a new in-context visual question answering dataset encompassing interleaved images and EHR data derived from MIMIC-IV and MIMIC-CXR-JPG databases. Results demonstrate the SOTA performance of MedPromptX, achieving an 11% improvement in F1-score compared to the baselines. Code and data are publicly available on https://github.com/BioMedIA-MBZUAI/MedPromptX.
Citation
M. A. Shaaban, A. Khan, and M. Yaqub, “MedPromptX: Grounded Multimodal Prompting for Chest X-Ray Diagnosis,” Lecture Notes in Computer Science, vol. 15401 LNCS, pp. 211–222, 2025, doi: 10.1007/978-3-031-84525-3_18
Source
Lecture Notes in Computer Science
Conference
International Conference on Medical Image Computing and Computer-Assisted InterventionMarrakesh, Morocco7 October 2024
Keywords
Few-shot Learning, Medical Diagnosis, Multimodal Large Language Models, Visual Grounding, Visual Question Answering
Subjects
Source
International Conference on Medical Image Computing and Computer-Assisted InterventionMarrakesh, Morocco7 October 2024
Publisher
Springer Nature