An Improved Knowledge Distillation Algorithm and Its Application to Object Detection
Min Yao1,*, Guofeng Liu2, Yaozu Zhang3, Guangjie Hu1
1 School of Information Engineering, Shanghai Maritime University, Shanghai, 201306, China
2 Baidu, Beijing, 100000, China
3 Shanghai Freesense Technology Co., Ltd., Shanghai, 200000, China
* Corresponding Author: Min Yao. Email: 
Computers, Materials & Continua https://doi.org/10.32604/cmc.2025.060609
Received 06 November 2024; Accepted 08 February 2025; Published online 07 March 2025
Abstract
Knowledge distillation (KD) is an emerging model compression technique for learning compact object detector models. Previous KD often focused solely on distilling from the logits layer or the feature intermediate layers, which may limit the comprehensive learning of the student network. Additionally, the imbalance between the foreground and background also affects the performance of the model. To address these issues, this paper employs feature-based distillation to enhance the detection performance of the bounding box localization part, and logit-based distillation to improve the detection performance of the category prediction part. Specifically, for the intermediate layer feature distillation, we introduce feature resampling to reduce the risk of the student model merely imitating the teacher model. At the same time, we incorporate a Spatial Attention Mechanism (SAM) to highlight the foreground features learned by the student model. In terms of output layer feature distillation, we divide the traditional distillation targets into target-class objects and non-target-class objects, aiming to improve overall distillation performance. Furthermore, we introduce a one-to-many matching distillation strategy based on Feature Alignment Module (FAM), which further enhances the student model’s feature representation ability, making its feature distribution closer to that of the teacher model, and thus demonstrating superior localization and classification capabilities in object detection tasks. Experimental results demonstrate that our proposed methodology outperforms conventional distillation techniques in terms of object detecting performance.
Keywords
Deep learning; model compression; knowledge distillation; object detection
Open Access
Login
Register
Submit a Paper
Propose a Special lssue
Download PDF
Downloads
Citation Tools
