IFHE: Intermediate-Feature Heterogeneity Enhancement for Image Synthesis in Data-Free Knowledge Distillation

Data-free knowledge distillation (DFKD) explores training a compact student network only by a pre-trained teacher without real data. Prevailing DFKD methods mainly consist of image synthesis and knowledge distillation. The synthesized images are crucial to enhance the student network performance. Ho...

Ausführliche Beschreibung

Gespeichert in:
Bibliographische Detailangaben
Veröffentlicht in:2023 60th ACM/IEEE Design Automation Conference (DAC) S. 1 - 6
Hauptverfasser: Chen, Yi, Liu, Ning, Ren, Ao, Yang, Tao, Liu, Duo
Format: Tagungsbericht
Sprache:Englisch
Veröffentlicht: IEEE 09.07.2023
Schlagworte:
Design automation
Image synthesis
Knowledge distillation
Knowledge engineering
model compression
Training
Online-Zugang:Volltext
Tags: Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
Abstract Data-free knowledge distillation (DFKD) explores training a compact student network only by a pre-trained teacher without real data. Prevailing DFKD methods mainly consist of image synthesis and knowledge distillation. The synthesized images are crucial to enhance the student network performance. However, the images synthesized by existing methods cause high homogeneity on intermediate features, incurring undesired distillation performance. To address this problem, we propose the Intermediate-Feature Heterogeneity Enhancement (IFHE) method, which effectively enhances the heterogeneity of synthesized images by minimizing the loss between intermediate features and pre-set labels of the synthesized images Our IFHE outperforms the SOTA results on CIFAR-10/100 datasets of representative networks.
AbstractList Data-free knowledge distillation (DFKD) explores training a compact student network only by a pre-trained teacher without real data. Prevailing DFKD methods mainly consist of image synthesis and knowledge distillation. The synthesized images are crucial to enhance the student network performance. However, the images synthesized by existing methods cause high homogeneity on intermediate features, incurring undesired distillation performance. To address this problem, we propose the Intermediate-Feature Heterogeneity Enhancement (IFHE) method, which effectively enhances the heterogeneity of synthesized images by minimizing the loss between intermediate features and pre-set labels of the synthesized images Our IFHE outperforms the SOTA results on CIFAR-10/100 datasets of representative networks.
Author Chen, Yi
Liu, Duo
Liu, Ning
Yang, Tao
Ren, Ao
Author_xml – sequence: 1
  givenname: Yi
  surname: Chen
  fullname: Chen, Yi
  email: chen_yi@cqu.edu.com
  organization: Chongqing University,School of Computer Science,Chongqing,China
– sequence: 2
  givenname: Ning
  surname: Liu
  fullname: Liu, Ning
  email: ningliu1220@gmail.com
  organization: Midea Group,Beijing,China
– sequence: 3
  givenname: Ao
  surname: Ren
  fullname: Ren, Ao
  email: ren.ao@cqu.edu.com
  organization: Chongqing University,School of Computer Science,Chongqing,China
– sequence: 4
  givenname: Tao
  surname: Yang
  fullname: Yang, Tao
  email: yangtao@cqu.edu.com
  organization: Chongqing University,School of Computer Science,Chongqing,China
– sequence: 5
  givenname: Duo
  surname: Liu
  fullname: Liu, Duo
  email: liuduo@cqu.edu.com
  organization: Chongqing University,School of Computer Science,Chongqing,China
BookMark eNo1kM1Kw0AUhUdQUGveQGReIHX-Msl1V9rGBAsu1HW56dy0A8lEkhHJ21tQVwcOH4ePc8suwxCIsQcpllIKeNys1pkFBUsllF5KoUyey_yCJZBDoTOhlTaFvGbJNPlGWJEVRlhzw7q6rLZPvA6Rxp6cx0hpSRi_RuIVncvhSIF8nPk2nDAcqKcQeTuMvO7xSPxtDvFEk5-4D3yDEdNyJOIvYfjuyJ2BjZ-i7zqMfgh37KrFbqLkLxfso9y-r6t09_pcr1e7FBWImOoDaMwUWjKNaQHo7EoFSesaZTSI3EibtbbRQMpha61yBw2tKpx1CDrXC3b_u-uJaP85-h7Hef9_iv4B4oZa9Q
ContentType Conference Proceeding
DBID 6IE
6IH
CBEJK
RIE
RIO
DOI 10.1109/DAC56929.2023.10247717
DatabaseName IEEE Electronic Library (IEL) Conference Proceedings
IEEE Proceedings Order Plan (POP) 1998-present by volume
IEEE Xplore All Conference Proceedings
IEEE Electronic Library (IEL)
IEEE Proceedings Order Plans (POP) 1998-present
DatabaseTitleList
Database_xml – sequence: 1
  dbid: RIE
  name: IEEE Electronic Library (IEL)
  url: https://ieeexplore.ieee.org/
  sourceTypes: Publisher
DeliveryMethod fulltext_linktorsrc
EISBN 9798350323481
EndPage 6
ExternalDocumentID 10247717
Genre orig-research
GrantInformation_xml – fundername: National Natural Science Foundation of China
  funderid: 10.13039/501100001809
GroupedDBID 6IE
6IH
ACM
ALMA_UNASSIGNED_HOLDINGS
CBEJK
RIE
RIO
ID FETCH-LOGICAL-a290t-3c93a52a6e4b4f99e840e8e16db2439074165f6b39e2daf662dc39f28d6da9373
IEDL.DBID RIE
ISICitedReferencesCount 0
ISICitedReferencesURI http://www.webofscience.com/api/gateway?GWVersion=2&SrcApp=Summon&SrcAuth=ProQuest&DestLinkType=CitingArticles&DestApp=WOS_CPL&KeyUT=001073487300052&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D
IngestDate Wed Aug 27 02:51:00 EDT 2025
IsPeerReviewed false
IsScholarly true
Language English
LinkModel DirectLink
MergedId FETCHMERGED-LOGICAL-a290t-3c93a52a6e4b4f99e840e8e16db2439074165f6b39e2daf662dc39f28d6da9373
PageCount 6
ParticipantIDs ieee_primary_10247717
PublicationCentury 2000
PublicationDate 2023-July-9
PublicationDateYYYYMMDD 2023-07-09
PublicationDate_xml – month: 07
  year: 2023
  text: 2023-July-9
  day: 09
PublicationDecade 2020
PublicationTitle 2023 60th ACM/IEEE Design Automation Conference (DAC)
PublicationTitleAbbrev DAC
PublicationYear 2023
Publisher IEEE
Publisher_xml – name: IEEE
SSID ssib060584064
Score 2.2248468
Snippet Data-free knowledge distillation (DFKD) explores training a compact student network only by a pre-trained teacher without real data. Prevailing DFKD methods...
SourceID ieee
SourceType Publisher
StartPage 1
SubjectTerms Design automation
Image synthesis
Knowledge distillation
Knowledge engineering
model compression
Training
Title IFHE: Intermediate-Feature Heterogeneity Enhancement for Image Synthesis in Data-Free Knowledge Distillation
URI https://ieeexplore.ieee.org/document/10247717
WOSCitedRecordID wos001073487300052&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D
hasFullText 1
inHoldings 1
isFullTextHit
isPrint
link http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV3PS8MwFA5uePCk4sTf5OA1s02atPEm28qGMgb-YLeRJi84mJ10neB_b5KtigcP3ppCU3gv6fv63vvyIXQNPNKKWU1Myg1JOBNERW5ojVGMSxrbIJ3w8pCOx9l0KidbsnrgwgBAaD6Drr8MtXyz1GufKnM7nCap-_9ooVaaphuyVrN4fHnPBadkywKOI3nTv-tx4cJ_10uEd5uHf8mohCiS7__z_Qeo88PHw5PvSHOIdqA8QotRPhzc4pDSC_yPGogHdOsK8NA3uSzd2gAHsvGgfPW-9ZNjh1Hx6M19RPDjZ-nA32q-wvMS91WtSF4B4PsmyYb7fvsvNr1yHfScD556Q7LVTiCKyqgmTEumOFUCkiKxUoIzDmQQe_koh0ECEONWFEwCNcoKQY1m0tLMCKMcZGHHqF0uSzhBOGGgJaRJZJWDd4WS1sEIGmtf4ctYwU9Rx5tq9r45HmPWWOnsj_vnaM87JPS8ygvUrqs1XKJd_VHPV9VVcOoXcpijrg
linkProvider IEEE
linkToHtml http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV3PS8MwFA46BT2pOPG3OXjtbPOjbbzJttKxOQZO2W2kzQsOZiddJ_jfm2Sr4sGDt6bQUN5L876-9758CN0C93NJde6piCuPcRp60jdDrZSkXJBAO-mEl0E0HMaTiRhtyOqOCwMArvkMWvbS1fLVIl_ZVJn5wgmLzP_HNtrhjJFgTdeql48t8JnwxDY84MAXd52HNg8NAGhZkfBW_fgvIRUXR5KDf77BIWr-MPLw6DvWHKEtKI7RvJek3XvsknqOAVKBZyHdqgSc2jaXhVkdYGA27hav1rt2cmxQKu69mW0EP30WBv4tZ0s8K3BHVtJLSgDcr9NsuGM3gPm6W66JnpPuuJ16G_UETxLhVx7NBZWcyBBYxrQQYIwDMQRWQMqgEAfFuA4zKoAoqcOQqJwKTWIVKmlACz1BjWJRwCnCjEIuIGK-lgbgZVJoAyRIkNsaX0wzfoaa1lTT9_UBGdPaSud_3L9Be-n4cTAd9Ib9C7RvneM6YMUlalTlCq7Qbv5RzZbltXPwFyTcpvU
openUrl ctx_ver=Z39.88-2004&ctx_enc=info%3Aofi%2Fenc%3AUTF-8&rfr_id=info%3Asid%2Fsummon.serialssolutions.com&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=proceeding&rft.title=2023+60th+ACM%2FIEEE+Design+Automation+Conference+%28DAC%29&rft.atitle=IFHE%3A+Intermediate-Feature+Heterogeneity+Enhancement+for+Image+Synthesis+in+Data-Free+Knowledge+Distillation&rft.au=Chen%2C+Yi&rft.au=Liu%2C+Ning&rft.au=Ren%2C+Ao&rft.au=Yang%2C+Tao&rft.date=2023-07-09&rft.pub=IEEE&rft.spage=1&rft.epage=6&rft_id=info:doi/10.1109%2FDAC56929.2023.10247717&rft.externalDocID=10247717