Label-Anticipated Event Disentanglement for Audio-Visual Video Parsing
Zhou, Jinxing Guo, Dan Mao, Yuxin Zhong, Yiran Chang, Xiaojun Wang, Meng
Zhou, Jinxing
Guo, Dan
Mao, Yuxin
Zhong, Yiran
Chang, Xiaojun
Wang, Meng
Supervisor
Department
Computer Vision
Embargo End Date
Type
Conference proceeding
Date
2025
License
Language
English
Collections
Research Projects
Organizational Units
Journal Issue
Abstract
Audio-Visual Video Parsing (AVVP) task aims to detect and temporally locate events within audio and visual modalities. Multiple events can overlap in the timeline, making identification challenging. While traditional methods usually focus on improving the early audio-visual encoders to embed more effective features, the decoding phase – crucial for final event classification, often receives less attention. We aim to advance the decoding phase and improve its interpretability. Specifically, we introduce a new decoding paradigm, label semantic-based projection (LEAP), that employs labels texts of event categories, each bearing distinct and explicit semantics, for parsing potentially overlapping events. LEAP works by iteratively projecting encoded latent features of audio/visual segments onto semantically independent label embeddings. This process, enriched by modeling cross-modal (audio/visual-label) interactions, gradually disentangles event semantics within video segments to refine relevant label embeddings, guaranteeing a more discriminative and interpretable decoding process. To facilitate the LEAP paradigm, we propose a semantic-aware optimization strategy, which includes a novel audio-visual semantic similarity loss function. This function leverages the Intersection over Union of audio and visual events (EIoU) as a novel metric to calibrate audio-visual similarities at the feature level, accommodating the varied event densities across modalities. Extensive experiments demonstrate the superiority of our method, achieving new state-of-the-art performance for AVVP and also enhancing the relevant audio-visual event localization task.
Citation
J. Zhou, D. Guo, Y. Mao, Y. Zhong, X. Chang, and M. Wang, “Label-Anticipated Event Disentanglement for Audio-Visual Video Parsing,” Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) , vol. 15068, pp. 35–51, Jan. 2025, doi: 10.1007/978-3-031-72684-2_3.
Source
Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Conference
European Conference on Computer Vision (ECCV)
Keywords
Audio-visual event localization, Audio-visual video parsing, Event disentanglement
Subjects
Source
European Conference on Computer Vision (ECCV)
Publisher
Springer Nature