Decoupling Clinical and Class-Agnostic Features for Reliable Few-Shot Adaptation Under Shift
Rahman, Umaima Imam, Raza Yaqub, Mohammad K. Mahapatra, Dwarikanath
Rahman, Umaima
Imam, Raza
Yaqub, Mohammad K.
Mahapatra, Dwarikanath
Author
Supervisor
Department
Computer Vision
Embargo End Date
Type
Conference proceeding
Date
2026
License
Language
English
Collections
Research Projects
Organizational Units
Journal Issue
Abstract
Medical vision-language models (VLMs) offer promise for clinical decision support, yet their reliability under distribution shifts remains a major concern for safe deployment. These models often learn task-agnostic correlations due to variability in imaging protocols and free-text reports, limiting their generalizability and increasing the risk of failure in real-world settings. We propose DRiFt, a structured feature decoupling framework that explicitly separates clinically relevant signals from task-agnostic noise using parameter-efficient tuning (LoRA) and learnable prompt tokens. To enhance cross-modal alignment and reduce uncertainty, we curate high-quality, clinically grounded image-text pairs by generating captions for a diverse medical dataset. Our approach improves in-distribution performance by +11.4% Top-1 accuracy and +3.3% Macro-F1 over prior prompt-based methods, while maintaining strong robustness across unseen datasets. Ablation studies reveal that disentangling task-relevant features and careful alignment significantly enhance model generalization and reduce unpredictable behavior under domain shift. These insights contribute toward building safer, more trustworthy VLMs for clinical use. The code is available at https://github.com/rumaima/DRiFt.
Citation
U. Rahman, R. Imam, M. Yaqub, and D. Mahapatra, “Decoupling Clinical and Class-Agnostic Features for Reliable Few-Shot Adaptation Under Shift,” pp. 123–133, 2026, doi: 10.1007/978-3-032-06593-3_12
Source
Lecture Notes in Computer Science
Conference
7th Workshop on Uncertainty for Safe Utilization of Machine Learning in Medical Imaging, UNSURE 2025, held in conjunction with 28th International Conference on Medical Image Computing and Computer Assisted Intervention, MICCAI 2025
Keywords
Distribution Shifts, Medical VLMs, OOD Generalization
Subjects
Source
7th Workshop on Uncertainty for Safe Utilization of Machine Learning in Medical Imaging, UNSURE 2025, held in conjunction with 28th International Conference on Medical Image Computing and Computer Assisted Intervention, MICCAI 2025
Publisher
Springer Nature