Model Compression Using Optimal Transport
Model compression methods are important to allow for easier deployment of deep learning models in compute, memory and energy-constrained environments such as mobile phones. Knowledge distillation is a class of model compression algorithms where knowledge from a large teacher network is transferred t...
Uloženo v:
| Vydáno v: | Proceedings / IEEE Workshop on Applications of Computer Vision s. 3645 - 3654 |
|---|---|
| Hlavní autoři: | , |
| Médium: | Konferenční příspěvek |
| Jazyk: | angličtina |
| Vydáno: |
IEEE
01.01.2022
|
| Témata: | |
| ISSN: | 2642-9381 |
| On-line přístup: | Získat plný text |
| Tagy: |
Přidat tag
Žádné tagy, Buďte první, kdo vytvoří štítek k tomuto záznamu!
|
| Abstract | Model compression methods are important to allow for easier deployment of deep learning models in compute, memory and energy-constrained environments such as mobile phones. Knowledge distillation is a class of model compression algorithms where knowledge from a large teacher network is transferred to a smaller student network thereby improving the student's performance. In this paper, we show how optimal transport-based loss functions can be used for training a student network which encourages learning student network parameters that help bring the distribution of student features closer to that of the teacher features. We present image classification results on CIFAR-100, SVHN and ImageNet and show that the proposed optimal transport loss functions perform comparably to or better than other loss functions. |
|---|---|
| AbstractList | Model compression methods are important to allow for easier deployment of deep learning models in compute, memory and energy-constrained environments such as mobile phones. Knowledge distillation is a class of model compression algorithms where knowledge from a large teacher network is transferred to a smaller student network thereby improving the student's performance. In this paper, we show how optimal transport-based loss functions can be used for training a student network which encourages learning student network parameters that help bring the distribution of student features closer to that of the teacher features. We present image classification results on CIFAR-100, SVHN and ImageNet and show that the proposed optimal transport loss functions perform comparably to or better than other loss functions. |
| Author | Lohit, Suhas Jones, Michael |
| Author_xml | – sequence: 1 givenname: Suhas surname: Lohit fullname: Lohit, Suhas email: slohit@merl.com organization: Mitsubishi Electric Research Laboratories,Cambridge,MA,USA – sequence: 2 givenname: Michael surname: Jones fullname: Jones, Michael email: mjones@merl.com organization: Mitsubishi Electric Research Laboratories,Cambridge,MA,USA |
| BookMark | eNotzjtPhEAUBeDRaOKy-gu0oLUA77y55Yb4StZsg1puBuZiMCxDZmj895JodZqTc76MXUxhIsbuOJScAz587uoPzZWuSgFClADSwhnLuDFaAXIN52wjjBIFyopfsSyl77WDHOWG3b8FT2Neh9McKaUhTPl7Gqav_DAvw8mNeRPdlOYQl2t22bsx0c1_blnz9NjUL8X-8Pxa7_bFIEAuhfItSG1B9JYQSfWd9MqtxxL7qnPKkmup7ZzxvrXVqiLtQIu28isIudyy27_ZgYiOc1wR8eeIFoyxKH8BBLFC-A |
| CODEN | IEEPAD |
| ContentType | Conference Proceeding |
| DBID | 6IE 6IL CBEJK RIE RIL |
| DOI | 10.1109/WACV51458.2022.00370 |
| DatabaseName | IEEE Electronic Library (IEL) Conference Proceedings IEEE Proceedings Order Plan All Online (POP All Online) 1998-present by volume IEEE Xplore All Conference Proceedings IEEE Electronic Library (IEL) IEEE Proceedings Order Plans (POP All) 1998-Present |
| DatabaseTitleList | |
| Database_xml | – sequence: 1 dbid: RIE name: IEEE/IET Electronic Library (IEL) url: https://ieeexplore.ieee.org/ sourceTypes: Publisher |
| DeliveryMethod | fulltext_linktorsrc |
| Discipline | Applied Sciences |
| EISBN | 1665409150 9781665409155 |
| EISSN | 2642-9381 |
| EndPage | 3654 |
| ExternalDocumentID | 9706679 |
| Genre | orig-research |
| GroupedDBID | 29G 29O 6IE 6IF 6IK 6IL 6IM 6IN AAJGR AAWTH ABLEC ADZIZ ALMA_UNASSIGNED_HOLDINGS BEFXN BFFAM BGNUA BKEBE BPEOZ CBEJK CHZPO IEGSK IPLJI M43 OCL RIE RIL RNS |
| ID | FETCH-LOGICAL-i203t-4db035702f7e99e4fc3d4a38139f8ca47eabebca6ddb78642e5a052b8d919913 |
| IEDL.DBID | RIE |
| ISICitedReferencesCount | 1 |
| ISICitedReferencesURI | http://www.webofscience.com/api/gateway?GWVersion=2&SrcApp=Summon&SrcAuth=ProQuest&DestLinkType=CitingArticles&DestApp=WOS_CPL&KeyUT=000800471203072&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D |
| IngestDate | Wed Aug 27 02:24:25 EDT 2025 |
| IsPeerReviewed | false |
| IsScholarly | true |
| Language | English |
| LinkModel | DirectLink |
| MergedId | FETCHMERGED-LOGICAL-i203t-4db035702f7e99e4fc3d4a38139f8ca47eabebca6ddb78642e5a052b8d919913 |
| PageCount | 10 |
| ParticipantIDs | ieee_primary_9706679 |
| PublicationCentury | 2000 |
| PublicationDate | 2022-Jan. |
| PublicationDateYYYYMMDD | 2022-01-01 |
| PublicationDate_xml | – month: 01 year: 2022 text: 2022-Jan. |
| PublicationDecade | 2020 |
| PublicationTitle | Proceedings / IEEE Workshop on Applications of Computer Vision |
| PublicationTitleAbbrev | WACV |
| PublicationYear | 2022 |
| Publisher | IEEE |
| Publisher_xml | – name: IEEE |
| SSID | ssj0039193 |
| Score | 2.2537966 |
| Snippet | Model compression methods are important to allow for easier deployment of deep learning models in compute, memory and energy-constrained environments such as... |
| SourceID | ieee |
| SourceType | Publisher |
| StartPage | 3645 |
| SubjectTerms | Computational modeling Computer vision Deep learning Deep Learning -> Efficient Training and Inference Methods for Networks Deep Learning; Object Detection/Recognition/Categorization; Statistical Methods; Learning and Optimization Image coding Knowledge engineering Mobile handsets Training |
| Title | Model Compression Using Optimal Transport |
| URI | https://ieeexplore.ieee.org/document/9706679 |
| WOSCitedRecordID | wos000800471203072&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D |
| hasFullText | 1 |
| inHoldings | 1 |
| isFullTextHit | |
| isPrint | |
| link | http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV09T8MwED21FQNTgRZB-ZAHFiRC3diO7RFVVAyodKigW2XHjlSppKhN-f34khAYWNisLJbPOt-L_d49gBurlEEgH5kAHiKuHUMj9yQyKRc8ybwwthQKP8vpVC0WetaCu0YL470vyWf-HoflW77bpHu8KhtqiZRM3Ya2lEml1fo-dZkOSKSWxo2oHr49jF8DFhDI3oqxJydDO-JfBipl_Zh0_zfzEfR_hHhk1pSYY2j5_AS6NXIkdV7uenCLlmZrgsld8VpzUnIByEs4Ed7NmjQ9zPswnzzOx09RbYIQrWLKiog7S5mQNM6k19rzLGWOm1Bnmc5Uarj0IZo2NYlzVqrwNxHCS0VsldPIamKn0Mk3uT8DQo3KnHJK6NjyJDZWjJjzUggT8nxk6Tn0cOHLj6rNxbJe8-DvzxdwiJGtbiMuoVNs9_4KDtLPYrXbXpd78wUWTI-5 |
| linkProvider | IEEE |
| linkToHtml | http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV1NTwIxEJ0gmugJFYzf9uDFxJXSj217NESCEZEDUW6k3XYTElwMLP5-290VPXjx1uyl6TTTedu-Nw_g2kipA5CPtAcPEVOWBiP3ONIJ4yxOHdemEAoPxHAoJxM1qsHtRgvjnCvIZ-4uDIu3fLtI1uGqrK1EoGSqLdjmjBFcqrW-z12qPBapxHEdrNpv991XjwZ44G-R0JWTBkPiXxYqRQXpNf439z60fqR4aLQpMgdQc9khNCrsiKrMXDXhJpiazVFI75LZmqGCDYBe_Jnwrudo08W8BePew7jbjyobhGhGMM0jZg2mXGCSCqeUY2lCLdO-0lKVykQz4Xw8TaJja42Q_n_CBxhzYqRVgddEj6CeLTJ3DAhrmVppJVfEsJhowzvUOsG59pneMfgEmmHh04-y0cW0WvPp35-vYLc_fh5MB4_DpzPYC1Eu7ybOoZ4v1-4CdpLPfLZaXhb79AXDYZMA |
| openUrl | ctx_ver=Z39.88-2004&ctx_enc=info%3Aofi%2Fenc%3AUTF-8&rfr_id=info%3Asid%2Fsummon.serialssolutions.com&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=proceeding&rft.title=Proceedings+%2F+IEEE+Workshop+on+Applications+of+Computer+Vision&rft.atitle=Model+Compression+Using+Optimal+Transport&rft.au=Lohit%2C+Suhas&rft.au=Jones%2C+Michael&rft.date=2022-01-01&rft.pub=IEEE&rft.eissn=2642-9381&rft.spage=3645&rft.epage=3654&rft_id=info:doi/10.1109%2FWACV51458.2022.00370&rft.externalDocID=9706679 |