IFHE: Intermediate-Feature Heterogeneity Enhancement for Image Synthesis in Data-Free Knowledge Distillation

Data-free knowledge distillation (DFKD) explores training a compact student network only by a pre-trained teacher without real data. Prevailing DFKD methods mainly consist of image synthesis and knowledge distillation. The synthesized images are crucial to enhance the student network performance. Ho...

Celý popis

Uloženo v:
Podrobná bibliografie
Vydáno v:2023 60th ACM/IEEE Design Automation Conference (DAC) s. 1 - 6
Hlavní autoři: Chen, Yi, Liu, Ning, Ren, Ao, Yang, Tao, Liu, Duo
Médium: Konferenční příspěvek
Jazyk:angličtina
Vydáno: IEEE 09.07.2023
Témata:
Design automation
Image synthesis
Knowledge distillation
Knowledge engineering
model compression
Training
On-line přístup:Získat plný text
Tagy: Přidat tag
Žádné tagy, Buďte první, kdo vytvoří štítek k tomuto záznamu!
Popis
Shrnutí:Data-free knowledge distillation (DFKD) explores training a compact student network only by a pre-trained teacher without real data. Prevailing DFKD methods mainly consist of image synthesis and knowledge distillation. The synthesized images are crucial to enhance the student network performance. However, the images synthesized by existing methods cause high homogeneity on intermediate features, incurring undesired distillation performance. To address this problem, we propose the Intermediate-Feature Heterogeneity Enhancement (IFHE) method, which effectively enhances the heterogeneity of synthesized images by minimizing the loss between intermediate features and pre-set labels of the synthesized images Our IFHE outperforms the SOTA results on CIFAR-10/100 datasets of representative networks.
DOI:10.1109/DAC56929.2023.10247717