Noise is an Efficient Learner for Zero-Shot Vision-Language Models
Imam, Raza Hanif, Asif Zhang, Jian Dawoud, Khaled Waleed Kementchedjhieva, Yova Yaqub, Mohammad
Imam, Raza
Hanif, Asif
Zhang, Jian
Dawoud, Khaled Waleed
Kementchedjhieva, Yova
Yaqub, Mohammad
Supervisor
Department
Computer Vision
Embargo End Date
Type
Conference proceeding
Date
License
Language
Collections
Research Projects
Organizational Units
Journal Issue
Abstract
Recently, test-time adaptation has garnered attention as a method for tuning models without labeled data. The conventional modus operandi for adapting pre-trained vision-language models (VLMs) during test-time primarily focuses on tuning learnable prompts; however, this approach over-looks potential distribution shifts in the visual representations themselves. In this work, we address this limitation by introducing Test-Time Noise Tuning (TNT), a novel method for handling unpredictable shifts in the visual space. TNT leverages, for the first time, a noise adaptation strategy that optimizes learnable noise directly in the visual input space, enabling adaptive feature learning from a single test sample. We further introduce a novel approach for inter-view representation alignment by explicitly enforcing coherence in embedding distances, ensuring consistent feature representations across views. Combined with scaled logits and confident view selection at inference, TNT sub-stantially enhances VLM generalization and calibration, achieving average gains of +7.38% on natural distributions benchmark and +0.80% on cross-dataset evaluations over zero-shot CLIP. These improvements lay a strong foundation for adaptive out-of-distribution handling.
Citation
R. Imam, A. Hanif, J. Zhang, K.W. Dawoud, Y. Kementchedjhieva, M. Yaqub, "Noise is an Efficient Learner for Zero-Shot Vision-Language Models," 2026, pp. 5879-5888.
Source
Conference
2025 IEEE/CVF International Conference on Computer Vision Workshops (ICCVW)
Keywords
46 Information and Computing Sciences, 4611 Machine Learning
Subjects
Source
2025 IEEE/CVF International Conference on Computer Vision Workshops (ICCVW)
Publisher
IEEE