Model Compression Using Optimal Transport

Model compression methods are important to allow for easier deployment of deep learning models in compute, memory and energy-constrained environments such as mobile phones. Knowledge distillation is a class of model compression algorithms where knowledge from a large teacher network is transferred t...

Full description

Saved in:
Bibliographic Details
Published in:Proceedings / IEEE Workshop on Applications of Computer Vision pp. 3645 - 3654
Main Authors: Lohit, Suhas, Jones, Michael
Format: Conference Proceeding
Language:English
Published: IEEE 01.01.2022
Subjects:
Computational modeling
Computer vision
Deep learning
Deep Learning -> Efficient Training and Inference Methods for Networks Deep Learning; Object Detection/Recognition/Categorization; Statistical Methods; Learning and Optimization
Image coding
Knowledge engineering
Mobile handsets
Training
ISSN:2642-9381
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Abstract Model compression methods are important to allow for easier deployment of deep learning models in compute, memory and energy-constrained environments such as mobile phones. Knowledge distillation is a class of model compression algorithms where knowledge from a large teacher network is transferred to a smaller student network thereby improving the student's performance. In this paper, we show how optimal transport-based loss functions can be used for training a student network which encourages learning student network parameters that help bring the distribution of student features closer to that of the teacher features. We present image classification results on CIFAR-100, SVHN and ImageNet and show that the proposed optimal transport loss functions perform comparably to or better than other loss functions.
AbstractList Model compression methods are important to allow for easier deployment of deep learning models in compute, memory and energy-constrained environments such as mobile phones. Knowledge distillation is a class of model compression algorithms where knowledge from a large teacher network is transferred to a smaller student network thereby improving the student's performance. In this paper, we show how optimal transport-based loss functions can be used for training a student network which encourages learning student network parameters that help bring the distribution of student features closer to that of the teacher features. We present image classification results on CIFAR-100, SVHN and ImageNet and show that the proposed optimal transport loss functions perform comparably to or better than other loss functions.
Author Lohit, Suhas
Jones, Michael
Author_xml – sequence: 1
  givenname: Suhas
  surname: Lohit
  fullname: Lohit, Suhas
  email: slohit@merl.com
  organization: Mitsubishi Electric Research Laboratories,Cambridge,MA,USA
– sequence: 2
  givenname: Michael
  surname: Jones
  fullname: Jones, Michael
  email: mjones@merl.com
  organization: Mitsubishi Electric Research Laboratories,Cambridge,MA,USA
BookMark eNotzjtPhEAUBeDRaOKy-gu0oLUA77y55Yb4StZsg1puBuZiMCxDZmj895JodZqTc76MXUxhIsbuOJScAz587uoPzZWuSgFClADSwhnLuDFaAXIN52wjjBIFyopfsSyl77WDHOWG3b8FT2Neh9McKaUhTPl7Gqav_DAvw8mNeRPdlOYQl2t22bsx0c1_blnz9NjUL8X-8Pxa7_bFIEAuhfItSG1B9JYQSfWd9MqtxxL7qnPKkmup7ZzxvrXVqiLtQIu28isIudyy27_ZgYiOc1wR8eeIFoyxKH8BBLFC-A
CODEN IEEPAD
ContentType Conference Proceeding
DBID 6IE
6IL
CBEJK
RIE
RIL
DOI 10.1109/WACV51458.2022.00370
DatabaseName IEEE Electronic Library (IEL) Conference Proceedings
IEEE Proceedings Order Plan All Online (POP All Online) 1998-present by volume
IEEE Xplore All Conference Proceedings
IEEE Xplore
IEEE Proceedings Order Plans (POP All) 1998-Present
DatabaseTitleList
Database_xml – sequence: 1
  dbid: RIE
  name: IEEE Xplore
  url: https://ieeexplore.ieee.org/
  sourceTypes: Publisher
DeliveryMethod fulltext_linktorsrc
Discipline Applied Sciences
EISBN 1665409150
9781665409155
EISSN 2642-9381
EndPage 3654
ExternalDocumentID 9706679
Genre orig-research
GroupedDBID 29G
29O
6IE
6IF
6IK
6IL
6IM
6IN
AAJGR
AAWTH
ABLEC
ADZIZ
ALMA_UNASSIGNED_HOLDINGS
BEFXN
BFFAM
BGNUA
BKEBE
BPEOZ
CBEJK
CHZPO
IEGSK
IPLJI
M43
OCL
RIE
RIL
RNS
ID FETCH-LOGICAL-i203t-4db035702f7e99e4fc3d4a38139f8ca47eabebca6ddb78642e5a052b8d919913
IEDL.DBID RIE
ISICitedReferencesCount 1
ISICitedReferencesURI http://www.webofscience.com/api/gateway?GWVersion=2&SrcApp=Summon&SrcAuth=ProQuest&DestLinkType=CitingArticles&DestApp=WOS_CPL&KeyUT=000800471203072&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D
IngestDate Wed Aug 27 02:24:25 EDT 2025
IsPeerReviewed false
IsScholarly true
Language English
LinkModel DirectLink
MergedId FETCHMERGED-LOGICAL-i203t-4db035702f7e99e4fc3d4a38139f8ca47eabebca6ddb78642e5a052b8d919913
PageCount 10
ParticipantIDs ieee_primary_9706679
PublicationCentury 2000
PublicationDate 2022-Jan.
PublicationDateYYYYMMDD 2022-01-01
PublicationDate_xml – month: 01
  year: 2022
  text: 2022-Jan.
PublicationDecade 2020
PublicationTitle Proceedings / IEEE Workshop on Applications of Computer Vision
PublicationTitleAbbrev WACV
PublicationYear 2022
Publisher IEEE
Publisher_xml – name: IEEE
SSID ssj0039193
Score 2.2537966
Snippet Model compression methods are important to allow for easier deployment of deep learning models in compute, memory and energy-constrained environments such as...
SourceID ieee
SourceType Publisher
StartPage 3645
SubjectTerms Computational modeling
Computer vision
Deep learning
Deep Learning -> Efficient Training and Inference Methods for Networks Deep Learning; Object Detection/Recognition/Categorization; Statistical Methods; Learning and Optimization
Image coding
Knowledge engineering
Mobile handsets
Training
Title Model Compression Using Optimal Transport
URI https://ieeexplore.ieee.org/document/9706679
WOSCitedRecordID wos000800471203072&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D
hasFullText 1
inHoldings 1
isFullTextHit
isPrint
link http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV07TwMxDLbaioGpQIugPJSBBYmj1yR3SUZUUTGg0qGCblUu8UmVyhX1we8nvjsKAwtDpChL5ES2vzifbYAbqTkK7jEiMBxJkwWVslxHudXc5y5WDsubflbjsZ7NzKQBd_tcGEQsyWd4T9PyL9-v3I5CZX2jiJJpmtBUKq1ytb6trjABidSpcYPY9N8ehq8BCyTE3uJUk1NQO-JfDVRK_zFq_2_nI-j-JOKxyd7FHEMDixNo18iR1Xq56cAttTRbMlLuitdasJILwF6CRXi3S7avYd6F6ehxOnyK6iYI0YLHYhtJn8UiUTHPFRqDMnfCSxv8rDC5dlYqtBkRmlLvM6XDawITGyc8094Qq0mcQqtYFXgGzHKR40A7GUCBdIbq9kidpmH4zCfKnkOHBJ9_VGUu5rXMvb-XL-CQTraKRlxCa7ve4RUcuM_tYrO-Lu_mC2H1j6s
linkProvider IEEE
linkToHtml http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV1NTwIxEJ0gmugJFYzf7sGLiStL26Xt0RAJRkQORLmRbjubkOBiYPH321lW9ODFQ5Oml6ZtZua1fTMP4Foohpw5DAkMh0In3qQMU2FqFHOpjaTF4qT7cjBQ47EeVuB2kwuDiAX5DO-oW_zlu7ld0VNZU0uiZOot2CblrDJb69vvcu2xSJkc14p08-2-8-rRQEz8LUZVOTkJEv-SUCkiSLf2v7n3ofGTihcMN0HmACqYHUKtxI5BaZnLOtyQqNksIPNeM1uzoGADBC_eJ7ybWbCpYt6AUfdh1OmFpQxCOGURz0PhkojHMmKpRK1RpJY7YXyk5TpV1giJJiFKU9u5RCp_n8DYRDFLlNPEa-JHUM3mGR5DYBhPsaWs8LBAWE2Ve4Rqt31ziYulOYE6LXzysS50MSnXfPr38BXs9kbP_Un_cfB0Bnu0y-u3iXOo5osVXsCO_cyny8VlcU5fEjqS9A
openUrl ctx_ver=Z39.88-2004&ctx_enc=info%3Aofi%2Fenc%3AUTF-8&rfr_id=info%3Asid%2Fsummon.serialssolutions.com&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=proceeding&rft.title=Proceedings+%2F+IEEE+Workshop+on+Applications+of+Computer+Vision&rft.atitle=Model+Compression+Using+Optimal+Transport&rft.au=Lohit%2C+Suhas&rft.au=Jones%2C+Michael&rft.date=2022-01-01&rft.pub=IEEE&rft.eissn=2642-9381&rft.spage=3645&rft.epage=3654&rft_id=info:doi/10.1109%2FWACV51458.2022.00370&rft.externalDocID=9706679