IFHE: Intermediate-Feature Heterogeneity Enhancement for Image Synthesis in Data-Free Knowledge Distillation

Data-free knowledge distillation (DFKD) explores training a compact student network only by a pre-trained teacher without real data. Prevailing DFKD methods mainly consist of image synthesis and knowledge distillation. The synthesized images are crucial to enhance the student network performance. Ho...

Full description

Saved in:
Bibliographic Details
Published in:2023 60th ACM/IEEE Design Automation Conference (DAC) pp. 1 - 6
Main Authors: Chen, Yi, Liu, Ning, Ren, Ao, Yang, Tao, Liu, Duo
Format: Conference Proceeding
Language:English
Published: IEEE 09.07.2023
Subjects:
Design automation
Image synthesis
Knowledge distillation
Knowledge engineering
model compression
Training
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:Data-free knowledge distillation (DFKD) explores training a compact student network only by a pre-trained teacher without real data. Prevailing DFKD methods mainly consist of image synthesis and knowledge distillation. The synthesized images are crucial to enhance the student network performance. However, the images synthesized by existing methods cause high homogeneity on intermediate features, incurring undesired distillation performance. To address this problem, we propose the Intermediate-Feature Heterogeneity Enhancement (IFHE) method, which effectively enhances the heterogeneity of synthesized images by minimizing the loss between intermediate features and pre-set labels of the synthesized images Our IFHE outperforms the SOTA results on CIFAR-10/100 datasets of representative networks.
DOI:10.1109/DAC56929.2023.10247717