Video Instance Segmentation in an Open-World
Thawakar, Omkar Narayan, Sanath Cholakkal, Hisham Muhammad Anwer, Rao Khan, Salman Laaksonen, Jorma Shah, Mubarak Khan, Fahad Shahbaz
Thawakar, Omkar
Narayan, Sanath
Cholakkal, Hisham
Muhammad Anwer, Rao
Khan, Salman
Laaksonen, Jorma
Shah, Mubarak
Khan, Fahad Shahbaz
Supervisor
Department
Machine Learning
Embargo End Date
Type
Journal article
Date
2025
License
Language
English
Collections
Research Projects
Organizational Units
Journal Issue
Abstract
Existing video instance segmentation (VIS) approaches generally follow a closed-world assumption, where only seen category instances are identified and spatio-temporally segmented at inference. Open-world formulation relaxes the close-world static-learning assumption as follows: (a) first, it distinguishes a set of known categories as well as labels an unknown object as ‘unknown’ and then (b) it incrementally learns the class of an unknown as and when the corresponding semantic labels become available. We propose the first open-world VIS approach, named OW-VISFormer, that introduces a novel feature enrichment mechanism and a spatio-temporal objectness (STO) module. The feature enrichment mechanism based on a light-weight auxiliary network aims at accurate pixel-level (unknown) object delineation from the background as well as distinguishing category-specific known semantic classes. The STO module strives to generate instance-level pseudo-labels by enhancing the foreground activations through a contrastive loss. Moreover, we also introduce an extensive experimental protocol to measure the characteristics of OW-VIS. Our OW-VISFormer performs favorably against a solid baseline in OW-VIS setting. Further, we evaluate our contributions in the standard fully-supervised VIS setting by integrating them into the recent SeqFormer, achieving an absolute gain of 1.6% AP on Youtube-VIS 2019 val. set. Lastly, we show the generalizability of our contributions for the open-world detection (OWOD) setting, outperforming the best existing OWOD method in the literature. Code, models along with OW-VIS splits are available at https://github.com/OmkarThawakar/OWVISFormer.
Citation
O. Thawakar et al., “Video Instance Segmentation in an Open-World,” Int J Comput Vis, vol. 133, no. 1, pp. 398–409, Jan. 2024, doi: 10.1007/S11263-024-02195-4/METRICS
Source
International Journal of Computer Vision
Conference
Keywords
Object-detection, Open-world segmentation, Transformers, Video instance segmentation, Video object detection
Subjects
Source
Publisher
Springer Nature