Efficient Video Object Segmentation via Modulated Cross-Attention Memory
Shaker, Abdelrahman Talal, Syed Danelljan, Martin Khan, Salman Yang, Ming-Hsuan Khan, Fahad Shahbaz
Shaker, Abdelrahman
Talal, Syed
Danelljan, Martin
Khan, Salman
Yang, Ming-Hsuan
Khan, Fahad Shahbaz
Supervisor
Department
Computer Vision
Embargo End Date
Type
Conference proceeding
Date
2025
License
Language
English
Collections
Research Projects
Organizational Units
Journal Issue
Abstract
Recently, transformer-based approaches have shown promising results for semi-supervised video object segmentation. However, these approaches typically struggle on long videos due to increased GPU memory demands, as they frequently expand the memory bank every few frames. We propose a transformer-based approach, named MAVOS, that introduces an optimized and dynamic long-term modulated cross-attention (MCA) memory to model temporal smoothness without requiring frequent memory expansion. The proposed MCA effectively encodes both local and global features at various levels of granularity while efficiently maintaining consistent speed regardless of the video length. Extensive experiments on multiple benchmarks, LVOS, Long-Time Video, and DAVIS 2017, demonstrate the effectiveness of our proposed contributions leading to real-time inference and markedly reduced memory demands without any degradation in segmentation accuracy on long videos. Compared to the best existing transformer-based approach, our MAVOS increases the speed by 7.6x, while significantly reducing the GPU memory by 87% with comparable segmentation performance on short and long video datasets. Notably, on the LVOS dataset, our MAVOS achieves a J&F score of 63.3% while operating at 37 frames per second (FPS) on a single V100 GPU. Our code and models will be publicly released at https://github.com/Amshaker/MAVOS.
Citation
A. Shaker, S. Talal, M. Danelljan, S. Khan, M. -H. Yang and F. S. Khan, "Efficient Video Object Segmentation via Modulated Cross-Attention Memory," 2025 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), Tucson, AZ, USA, 2025, pp. 8681-8690, doi: 10.1109/WACV61041.2025.00841
Source
Proceedings - 2025 IEEE Winter Conference on Applications of Computer Vision, WACV 2025
Conference
2025 IEEE/CVF Winter Conference on Applications of Computer Vision, WACV 2025
Keywords
Visual object segmentation, efficient long-term memory, transformer-based VOS
Subjects
Source
2025 IEEE/CVF Winter Conference on Applications of Computer Vision, WACV 2025
Publisher
IEEE