O-TPT: Orthogonality Constraints for Calibrating Test-time Prompt Tuning in Vision-Language Models
Sharifdeen, Ashshak Munir, Muhammad Akhtar Baliah, Sanoojan Khan, Salman Khan, Muhammad Haris
Sharifdeen, Ashshak
Munir, Muhammad Akhtar
Baliah, Sanoojan
Khan, Salman
Khan, Muhammad Haris
Supervisor
Department
Computer Vision
Embargo End Date
Type
Conference proceeding
Date
2025
License
Language
English
Collections
Research Projects
Organizational Units
Journal Issue
Abstract
Test-time prompt tuning for vision-language models (VLMs) is getting attention because of their ability to learn with unlabeled data without fine-tuning. Although test-time prompt tuning methods for VLMs can boost accuracy, the resulting models tend to demonstrate poor calibration, which casts doubts on the reliability and trustworthiness of these models. Notably, more attention needs to be devoted to calibrating the test-time prompt tuning in vision-language models. To this end, we propose a new approach, called O-TPT that introduces orthogonality constraints on the textual features corresponding to the learnable prompts for calibrating test-time prompt tuning in VLMs. Towards introducing orthogonality constraints, we make the following contributions. First, we uncover new insights behind the suboptimal calibration performance of existing methods relying on textual feature dispersion. Second, we show that imposing a simple orthogonalization of textual features is a more effective approach towards obtaining textual dispersion. We conduct extensive experiments on various datasets with different backbones and baselines. The results indicate that our method consistently outperforms the prior state of the art in significantly reducing the overall average calibration error. Also, our method surpasses the zero-shot calibration performance on fine-grained classification tasks. Our code is available at https://github.com/ashshaksharifdeen/O-TPT.
Citation
A. Sharifdeen, M. A. Munir, S. Baliah, S. Khan and M. H. Khan, "O-TPT: Orthogonality Constraints for Calibrating Test-time Prompt Tuning in Vision-Language Models," 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Nashville, TN, USA, 2025, pp. 19942-19951, doi: 10.1109/CVPR52734.2025.01857.
Source
Proceedings of the Conference on Computer Vision and Pattern Recognition (CVPR)
Conference
Conference on Computer Vision and Pattern Recognition (CVPR)
Keywords
Calibration, Test-time prompt tuning, Clip, Vision-language models, Orthogonality
Subjects
Source
Conference on Computer Vision and Pattern Recognition (CVPR)
Publisher
IEEE