DPA: Dual Prototypes Alignment for Unsupervised Adaptation of Vision-Language Models
Ali, Eman Silva, Sathira Khan, Muhammad Haris
Ali, Eman
Silva, Sathira
Khan, Muhammad Haris
Supervisor
Department
Computer Vision
Embargo End Date
Type
Conference proceeding
Date
2025
License
Language
English
Collections
Research Projects
Organizational Units
Journal Issue
Abstract
Vision-language models (VLMs), e.g., CLIP, have shown remarkable potential in zero-shot image classification. However, adapting these models to new domains remains challenging, especially in unsupervised settings where labeled data is unavailable. Recent research has proposed pseudo-labeling approaches to adapt CLIP in an unsupervised manner using unlabeled target data. Nonetheless, these methods struggle due to noisy pseudo-labels resulting from the misalignment between CLIP's visual and textual representations. This study introduces DPA, an unsupervised domain adaptation method for VLMs. DPA introduces the concept of dual prototypes, acting as distinct classifiers, along with the convex combination of their outputs, thereby leading to accurate pseudo-label construction. Next, it ranks pseudo-labels to facilitate robust self-training, particularly during early training. Finally, it addresses visual-textual misalignment by aligning textual prototypes with image prototypes to further improve the adaptation performance. Experiments on 13 downstream vision tasks demonstrate that DPA significantly outperforms zero-shot CLIP and the state-of-the-art unsupervised adaptation baselines.
Citation
E. Ali, S. Silva and M. H. Khan, "DPA: Dual Prototypes Alignment for Unsupervised Adaptation of Vision-Language Models," 2025 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), Tucson, AZ, USA, 2025, pp. 6083-6093, doi: 10.1109/WACV61041.2025.00593
Source
Proceedings - 2025 IEEE Winter Conference on Applications of Computer Vision, WACV 2025
Conference
2025 IEEE/CVF Winter Conference on Applications of Computer Vision, WACV 2025
Keywords
Subjects
Source
2025 IEEE/CVF Winter Conference on Applications of Computer Vision, WACV 2025
Publisher
IEEE