Loading...
Can Language-Guided Unsupervised Adaptation Improve Medical Image Classification Using Unpaired Images and Texts?
Rahman, Umaima Imam, Raza Yaqub, Mohammad Ben Amor, Boulbaba Mahapatra, Dwarikanath
Rahman, Umaima
Imam, Raza
Yaqub, Mohammad
Ben Amor, Boulbaba
Mahapatra, Dwarikanath
Supervisor
Department
Computer Vision
Embargo End Date
Type
Conference proceeding
Date
2025
License
Language
English
Collections
Research Projects
Organizational Units
Journal Issue
Abstract
In medical image classification, supervised learning is challenging due to the scarcity of labeled medical images. To address this, we leverage the visual-textual alignment within Vision-Language Models (VLMs) to enable unsupervised learning of a medical image classifier. In this work, we propose Medical Unsupervised Adaptation (MedUnA) of VLMs, where the LLM-generated descriptions for each class are encoded into text embeddings and matched with class labels via a cross-modal adapter. This adapter attaches to a visual encoder of MedCLIP and aligns the visual embeddings through unsupervised learning, driven by a contrastive entropy-based loss and prompt tuning. Thereby, improving performance in scenarios where textual information is more abundant than labeled images, particularly in the healthcare domain. Unlike traditional VLMs, MedUnA uses unpaired images and text for learning representations and enhances the potential of VLMs beyond traditional constraints. We evaluate the performance on three chest X-ray datasets and two multiclass datasets (diabetic retinopathy and skin lesions), showing significant accuracy gains over the zero-shot baseline. Our code is available at https://github.com/rumaima/meduna.
Citation
U. Rahman, R. Imam, M. Yaqub, B. B. Amor and D. Mahapatra, "Can Language-Guided Unsupervised Adaptation Improve Medical Image Classification Using Unpaired Images and Texts?," 2025 IEEE 22nd International Symposium on Biomedical Imaging (ISBI), Houston, TX, USA, 2025, pp. 1-5, doi: 10.1109/ISBI60581.2025.10981057
Source
2025 IEEE 22nd International Symposium on Biomedical Imaging (ISBI)
Conference
IEEE International Symposium on Biomedical Imaging, 2025
Keywords
VLMs, Unpaired images and texts, Label-free tuning, Unsupervised learning, Prompt tuning
Subjects
Source
IEEE International Symposium on Biomedical Imaging, 2025
Publisher
IEEE