SPARTA: Spectral Prompt Agnostic Adversarial Attack on Medical Vision-Language Models
Hanif, Asif Zaheer, Zaigham Khan, Salman Hameed Khan, Fahad Shahbaz Muhammad Anwer, Rao Muhammad
Hanif, Asif
Zaheer, Zaigham
Khan, Salman Hameed
Khan, Fahad Shahbaz
Muhammad Anwer, Rao Muhammad
Supervisor
Department
Computer Vision
Embargo End Date
Type
Conference proceeding
Date
2026
License
Language
English
Collections
Research Projects
Organizational Units
Journal Issue
Abstract
Medical Vision-Language Models (Med-VLMs) are gaining popularity in different medical tasks, such as visual question-answering (VQA), captioning, and diagnosis support. However, despite their impressive performance, Med-VLMs remain vulnerable to adversarial attacks, much like their general-purpose counterparts. In this work, we investigate the cross-prompt transferability of adversarial attacks on Med-VLMs in the context of VQA. To this end, we propose a novel adversarial attack algorithm that operates in the frequency domain of images and employs a learnable text context within a max-min competitive optimization framework, enabling the generation of adversarial perturbations that are transferable across diverse prompts. Evaluation on three Med-VLMs and four Med-VQA datasets shows that our approach outperforms the baseline, achieving an average attack success rate of 67% (compared to baseline’s 62%).
Citation
A. Hanif, Z. Zaheer, S. Khan, F. S. Khan, and R. Anwer, “SPARTA: Spectral Prompt Agnostic Adversarial Attack on Medical Vision-Language Models,” pp. 69–80, 2026, doi: 10.1007/978-3-032-06593-3_7
Source
Lecture Notes in Computer Science
Conference
7th Workshop on Uncertainty for Safe Utilization of Machine Learning in Medical Imaging, UNSURE 2025, held in conjunction with 28th International Conference on Medical Image Computing and Computer Assisted Intervention, MICCAI 2025
Keywords
Adversarial Attack, Spectral Attack, Transferability, Vision-Language Models, Visual Question Answering
Subjects
Source
7th Workshop on Uncertainty for Safe Utilization of Machine Learning in Medical Imaging, UNSURE 2025, held in conjunction with 28th International Conference on Medical Image Computing and Computer Assisted Intervention, MICCAI 2025
Publisher
Springer Nature