Matching DNN Compression and Cooperative Training with Resources and Data Availability

To make machine learning (ML) sustainable and apt to run on the diverse devices where relevant data is, it is essential to compress ML models as needed, while still meeting the required learning quality and time performance. However, how much and when an ML model should be compressed, and where its...

Full description

Saved in:
Bibliographic Details
Published in:Annual Joint Conference of the IEEE Computer and Communications Societies pp. 1 - 10
Main Authors: Malandrino, F., Giacomo, G. Di, Karamzade, A., Levorato, M., Chiasserini, C. F.
Format: Conference Proceeding
Language:English
Published: IEEE 17.05.2023
Subjects:
Adaptation models
Approximation algorithms
Costs
Energy consumption
Heuristic algorithms
Prediction algorithms
Training
ISSN:2641-9874
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Abstract To make machine learning (ML) sustainable and apt to run on the diverse devices where relevant data is, it is essential to compress ML models as needed, while still meeting the required learning quality and time performance. However, how much and when an ML model should be compressed, and where its training should be executed, are hard decisions to make, as they depend on the model itself, the resources of the available nodes, and the data such nodes own. Existing studies focus on each of those aspects individually, however, they do not account for how such decisions can be made jointly and adapted to one another. In this work, we model the network system focusing on the training of DNNs, formalize the above multi-dimensional problem, and, given its NP-hardness, formulate an approximate dynamic programming problem that we solve through the PACT algorithmic framework. Importantly, PACT leverages a time-expanded graph representing the learning process, and a data-driven and theoretical approach for the prediction of the loss evolution to be expected as a consequence of training decisions. We prove that PACT's solutions can get as close to the optimum as desired, at the cost of an increased time complexity, and that, in any case, such complexity is polynomial. Numerical results also show that, even under the most disadvantageous settings, PACT outperforms state-of-the-art alternatives and closely matches the optimal energy cost.
AbstractList To make machine learning (ML) sustainable and apt to run on the diverse devices where relevant data is, it is essential to compress ML models as needed, while still meeting the required learning quality and time performance. However, how much and when an ML model should be compressed, and where its training should be executed, are hard decisions to make, as they depend on the model itself, the resources of the available nodes, and the data such nodes own. Existing studies focus on each of those aspects individually, however, they do not account for how such decisions can be made jointly and adapted to one another. In this work, we model the network system focusing on the training of DNNs, formalize the above multi-dimensional problem, and, given its NP-hardness, formulate an approximate dynamic programming problem that we solve through the PACT algorithmic framework. Importantly, PACT leverages a time-expanded graph representing the learning process, and a data-driven and theoretical approach for the prediction of the loss evolution to be expected as a consequence of training decisions. We prove that PACT's solutions can get as close to the optimum as desired, at the cost of an increased time complexity, and that, in any case, such complexity is polynomial. Numerical results also show that, even under the most disadvantageous settings, PACT outperforms state-of-the-art alternatives and closely matches the optimal energy cost.
Author Levorato, M.
Malandrino, F.
Karamzade, A.
Giacomo, G. Di
Chiasserini, C. F.
Author_xml – sequence: 1
  givenname: F.
  surname: Malandrino
  fullname: Malandrino, F.
  organization: CNR-IEIIT,Italy
– sequence: 2
  givenname: G. Di
  surname: Giacomo
  fullname: Giacomo, G. Di
  organization: Politecnico di Torino,Italy
– sequence: 3
  givenname: A.
  surname: Karamzade
  fullname: Karamzade, A.
  organization: UC Irvine,USA
– sequence: 4
  givenname: M.
  surname: Levorato
  fullname: Levorato, M.
  organization: UC Irvine,USA
– sequence: 5
  givenname: C. F.
  surname: Chiasserini
  fullname: Chiasserini, C. F.
  organization: Politecnico di Torino,Italy
BookMark eNo1kM1OwkAURkejiYC8gYvGfXHuTKfTWZIiSgIlMeiW3JneypjSkk7F8Pb-r06-5ORbnCG7aNqGGLsFPgHg5m5RzNf5eqWkkWYiuJAT4EIYrtMzNjbaZFJxKRNIxDkbiDSB2GQ6uWLDEN4455kW6YC9rLB3O9-8RrOiiPJ2f-goBN82ETbl124P1GHvjxRtOvTNt_jh-130RKF97xyFH2-GPUbTI_oara99f7pmlxXWgcZ_HLHn-f0mf4yX64dFPl3GHsD0MVRcQSl45lCSI22VK6XOjLMpgZGCo3YCM5SqBJCpsqlCslZDUhEKa-SI3fz-eiLaHjq_x-60_e8gPwFpTlXg
ContentType Conference Proceeding
DBID 6IE
6IH
CBEJK
RIE
RIO
DOI 10.1109/INFOCOM53939.2023.10229076
DatabaseName IEEE Electronic Library (IEL) Conference Proceedings
IEEE Proceedings Order Plan (POP) 1998-present by volume
IEEE Xplore All Conference Proceedings
IEEE Electronic Library (IEL)
IEEE Proceedings Order Plans (POP) 1998-present
DatabaseTitleList
Database_xml – sequence: 1
  dbid: RIE
  name: IEEE Electronic Library (IEL)
  url: https://ieeexplore.ieee.org/
  sourceTypes: Publisher
DeliveryMethod fulltext_linktorsrc
Discipline Engineering
EISBN 9798350334142
EISSN 2641-9874
EndPage 10
ExternalDocumentID 10229076
Genre orig-research
GrantInformation_xml – fundername: Horizon Europe
  funderid: 10.13039/100018693
GroupedDBID 6IE
6IF
6IH
6IK
6IL
6IM
6IN
AAJGR
AAWTH
ABLEC
ACGFS
ADZIZ
ALMA_UNASSIGNED_HOLDINGS
BEFXN
BFFAM
BGNUA
BKEBE
BPEOZ
CBEJK
CHZPO
IEGSK
IJVOP
IPLJI
M43
OCL
RIE
RIL
RIO
ID FETCH-LOGICAL-i119t-1f051d208ca3ece7b5cd3789cb6e19320a7c2a8a35d11365b65aebb714fea2b93
IEDL.DBID RIE
IngestDate Wed Aug 27 02:55:18 EDT 2025
IsPeerReviewed false
IsScholarly true
Language English
LinkModel DirectLink
MergedId FETCHMERGED-LOGICAL-i119t-1f051d208ca3ece7b5cd3789cb6e19320a7c2a8a35d11365b65aebb714fea2b93
PageCount 10
ParticipantIDs ieee_primary_10229076
PublicationCentury 2000
PublicationDate 2023-May-17
PublicationDateYYYYMMDD 2023-05-17
PublicationDate_xml – month: 05
  year: 2023
  text: 2023-May-17
  day: 17
PublicationDecade 2020
PublicationTitle Annual Joint Conference of the IEEE Computer and Communications Societies
PublicationTitleAbbrev INFOCOM
PublicationYear 2023
Publisher IEEE
Publisher_xml – name: IEEE
SSID ssj0008726
Score 2.2911665
Snippet To make machine learning (ML) sustainable and apt to run on the diverse devices where relevant data is, it is essential to compress ML models as needed, while...
SourceID ieee
SourceType Publisher
StartPage 1
SubjectTerms Adaptation models
Approximation algorithms
Costs
Energy consumption
Heuristic algorithms
Prediction algorithms
Training
Title Matching DNN Compression and Cooperative Training with Resources and Data Availability
URI https://ieeexplore.ieee.org/document/10229076
hasFullText 1
inHoldings 1
isFullTextHit
isPrint
link http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV07b8IwELYK6tAufVH1LQ9dA3Ecx_ZYQVErlcBAERuynYuEVAUEAan_vrYTaDt06ObYiRT5db7v_H2H0CNzVstO3ACI9U1iASJQcRIHQrM8kyqPtQfcJm88TcV0Kkc1Wd1zYQDAXz6Dtiv6WH62MBsHlXWcd2KduaSBGpwnFVlrv-0KHiW1qigJZec17Q-7wwGj0vNRItreff0rj4o3I_2Tf_7AKWp9E_LwaG9qztABFOfo-IeW4AWaDOym6uAk3EtT7JZ5dcO1wKrI7PNiCZXINx7XWSGww2DxDr9f-_d6qlT4aavmH5V-92cLvfefx92XoE6aEMwJkWVAcrvMsigURlEwwDUzGeVCGp2AO6yFiptICUVZ5tK5MJ0wBVpzEuegIi3pJWoWiwKuEOZE2laR2JOtC6-FOle5iHRIcyWcyuQ1arkumi0rXYzZrndu_qi_RUduIFzsnfA71CxXG7hHh2ZbzterBz-aX90Cn_4
linkProvider IEEE
linkToHtml http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV3NT8IwFG8UTdSLXxi_7cHrYN3WtT0akECEwQEJN9Jub8kSMwgMEv97226gHjx469otWfr1-n6vv99D6Jkaq6UnrgNE-yYBB-7IIAwcrmiaCJkGygJukz6LIj6dilFFVrdcGACwl8-gYYo2lp_M47WByprGO9HOXLiPDmgQeG5J19ptvJx5YaUrSlzR7EWdYWs4oL6wjBTPb2y__5VJxRqSzuk_f-EM1b8peXi0MzbnaA_yC3TyQ03wEk0Gels1gBJuRxE2C72845pjmSf6eb6AUuYbj6u8ENigsHiL4K_se21ZSPyykdlHqeD9WUfvnddxq-tUaROcjBBROCTVCy3xXB5LH2JgisaJz7iIVQjmuOZKFnuSS58mJqELVSGVoBQjQQrSU8K_QrV8nsM1wowI3cpDfbY1ATZXpTLlnnL9VHKjM3mD6qaLZotSGWO27Z3bP-qf0FF3POjP-r3o7Q4dm0ExkXjC7lGtWK7hAR3GmyJbLR_tyH4BXn2jRQ
openUrl ctx_ver=Z39.88-2004&ctx_enc=info%3Aofi%2Fenc%3AUTF-8&rfr_id=info%3Asid%2Fsummon.serialssolutions.com&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=proceeding&rft.title=Annual+Joint+Conference+of+the+IEEE+Computer+and+Communications+Societies&rft.atitle=Matching+DNN+Compression+and+Cooperative+Training+with+Resources+and+Data+Availability&rft.au=Malandrino%2C+F.&rft.au=Giacomo%2C+G.+Di&rft.au=Karamzade%2C+A.&rft.au=Levorato%2C+M.&rft.date=2023-05-17&rft.pub=IEEE&rft.eissn=2641-9874&rft.spage=1&rft.epage=10&rft_id=info:doi/10.1109%2FINFOCOM53939.2023.10229076&rft.externalDocID=10229076