Distilling Object Detectors via Monte Carlo Dropout
Yi, Junfei Zhang, Hui Mao, Jianxu Liu, Tengfei Li, Mingjie Lin, Sihao Gu, Hanyu Li, Zhihui Chang, Xiaojun Wang, Yaonan
Yi, Junfei
Zhang, Hui
Mao, Jianxu
Liu, Tengfei
Li, Mingjie
Lin, Sihao
Gu, Hanyu
Li, Zhihui
Chang, Xiaojun
Wang, Yaonan
Supervisor
Department
Computer Vision
Embargo End Date
Type
Journal article
Date
License
Language
English
Collections
Research Projects
Organizational Units
Journal Issue
Abstract
Knowledge distillation (KD) has become a fundamental technique for model compression in object detection tasks. The data noise and training randomness may cause the knowledge of the teacher model to be unreliable, referred to as knowledge uncertainty. Existing methods neglect this uncertainty, potentially hindering the student's capacity to capture and understand latent "dark knowledge". In this work, we introduce a novel strategy that explicitly incorporates knowledge uncertainty, named Uncertainty-Driven Knowledge Extraction and Transfer (UET). Given the unknown, high-dimensional nature of the knowledge distribution, we employ Monte Carlo dropout to effectively estimate the teacher's uncertainty. Leveraging information theory, we combine uncertainty with deterministic knowledge, enabling the student to benefit from both precision and diversity. UET is a plug-and-play method that integrates seamlessly with existing distillation techniques. We validate our approach through comprehensive experiments across various distillation strategies, detectors, and backbones. Specifically, UET achieves state-of-the-art results, with a ResNet50-based GFL detector obtaining 44.1% mAP on the COCO dataset-surpassing baseline performance by 3.9%.
Citation
J. Yi, H. Zhang, J. Mao, T. Liu, M. Li, S. Lin , et al., "Distilling Object Detectors via Monte Carlo Dropout," IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. PP, no. 99, pp. 1-15, 2026, https://doi.org/10.1109/tpami.2026.3674980.
Source
IEEE Transactions on Pattern Analysis and Machine Intelligence
Conference
Keywords
46 Information and Computing Sciences, 4605 Data Management and Data Science
Subjects
Source
Publisher
IEEE