CAE-DFKD: Bridging the Transferability Gap in Data-Free Knowledge Distillation

Data-Free Knowledge Distillation (DFKD) enables the knowledge transfer from the given pre-trained teacher network to the target student model without access to the real training data. Existing DFKD methods focus primarily on improving image recognition performance on associated datasets, often negle...

Celý popis

Uloženo v:
Podrobná bibliografie
Vydáno v:2025 62nd ACM/IEEE Design Automation Conference (DAC) s. 1 - 7
Hlavní autoři: Zhang, Zherui, Wang, Changwei, Xu, Rongtao, Xu, Wenhao, Xu, Shibiao, Zhang, Yu, Zhou, Jie, Guo, Li
Médium: Konferenční příspěvek
Jazyk:angličtina
Vydáno: IEEE 22.06.2025
Témata:
Contrastive learning
Design automation
Generators
Image recognition
Knowledge Distillation
Knowledge engineering
Knowledge transfer
Representation learning
Training
Training data
Transfer learning
On-line přístup:Získat plný text
Tagy: Přidat tag
Žádné tagy, Buďte první, kdo vytvoří štítek k tomuto záznamu!
Popis
Shrnutí:Data-Free Knowledge Distillation (DFKD) enables the knowledge transfer from the given pre-trained teacher network to the target student model without access to the real training data. Existing DFKD methods focus primarily on improving image recognition performance on associated datasets, often neglecting the crucial aspect of the transferability of learned representations. In this paper, we propose Category-Aware Embedding Data-Free Knowledge Distillation (CAE-DFKD), which addresses at the embedding level the limitations of previous rely on image-level methods to improve model generalization but fail when directly applied to DFKD. The superiority and flexibility of CAE-DFKD are extensively evaluated, including: i.) Significant efficiency advantages resulting from altering the generator training paradigm; ii.) Competitive performance with existing DFKD state-of-the-art methods on image recognition tasks; iii.) Remarkable transferability of data-free learned representations demonstrated in downstream tasks.
DOI:10.1109/DAC63849.2025.11132975