Enhancing Novel Object Detection via Cooperative Foundational Models
Bharadwaj, Rohit
Bharadwaj, Rohit
Author
Supervisor
Department
Machine Learning
Embargo End Date
2024-01-01
Type
Thesis
Date
2024
License
Language
English
Collections
Research Projects
Organizational Units
Journal Issue
Abstract
In this work, we address the challenging and emergent problem of novel object detection (NOD), focusing on the accurate detection of both known and novel object categories during inference. Traditional object detection algorithms are inherently closed set, limiting their capability to handle NOD. We present a novel approach to transform existing closed-set detectors into open-set detectors. This transformation is achieved by leveraging the complementary strengths of pre-trained foundational models, specifically CLIP and SAM, through our cooperative mechanism. Furthermore, by integrating this mechanism with state-of-the-art open-set detectors such as GDINO, we establish new benchmarks in object detection performance. Our method achieves 17.42 mAP in novel object detection and 42.08 mAP for known objects on the challenging LVIS dataset. Adapting our approach to the COCO OVD split, we surpass the current stateof- the-art by a margin of 7.2 AP50 for novel classes. Our code is available at https: //github.com/rohit901/cooperative-foundational-models.
Citation
R. Bharadwaj, "Enhancing Novel Object Detection via Cooperative Foundational Models", MS. Thesis, Machine Learning, MBZUAI, Abu Dhabi, UAE, 2024