DiffGAP: A Lightweight Diffusion Module in Contrastive Space for Bridging Cross-Model Gap
Mo, Shentong Chen, Zehua Bao, Fan Zhu, Jun
Mo, Shentong
Chen, Zehua
Bao, Fan
Zhu, Jun
Author
Supervisor
Department
Machine Learning
Embargo End Date
Type
Conference proceeding
Date
2025
License
Language
English
Collections
Research Projects
Organizational Units
Journal Issue
Abstract
Recent works in cross-modal understanding and generation, notably through models like CLAP (Contrastive Language-Audio Pretraining) and CAVP (Contrastive Audio-Visual Pretraining), have significantly enhanced the alignment of text, video, and audio embeddings via a single contrastive loss. However, these methods often overlook the bidirectional interactions and inherent noises present in each modality, which can crucially impact the quality and efficacy of cross-modal integration. To address this limitation, we introduce DiffGAP, a novel approach incorporating a lightweight generative module within the contrastive space. Specifically, our DiffGAP employs a bidirectional diffusion process tailored to bridge the cross-modal gap more effectively. This involves a denoising process on text and video embeddings conditioned on audio embeddings and vice versa, thus facilitating a more nuanced and robust cross-modal interaction. Our experimental results on VGGSound and AudioCaps datasets demonstrate that DiffGAP significantly improves performance in video/text-audio generation and retrieval tasks, confirming its effectiveness in enhancing cross-modal understanding and generation capabilities.
Citation
S. Mo, Z. Chen, F. Bao and J. Zhu, "DiffGAP: A Lightweight Diffusion Module in Contrastive Space for Bridging Cross-Model Gap," ICASSP 2025 - 2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Hyderabad, India, 2025, pp. 1-5, doi: 10.1109/ICASSP49660.2025.10889139
Source
ICASSP 2025 - 2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)
Conference
International Conference on Acoustics, Speech, and Signal Processing (ICASSP), 2025
Keywords
Video-to-Audio Generation, Text-to-Audio Generation, Video/Text-to-Audio Retrieval
Subjects
Source
International Conference on Acoustics, Speech, and Signal Processing (ICASSP), 2025
Publisher
IEEE