Matching DNN Compression and Cooperative Training with Resources and Data Availability
To make machine learning (ML) sustainable and apt to run on the diverse devices where relevant data is, it is essential to compress ML models as needed, while still meeting the required learning quality and time performance. However, how much and when an ML model should be compressed, and where its...
Gespeichert in:
| Veröffentlicht in: | Annual Joint Conference of the IEEE Computer and Communications Societies S. 1 - 10 |
|---|---|
| Hauptverfasser: | , , , , |
| Format: | Tagungsbericht |
| Sprache: | Englisch |
| Veröffentlicht: |
IEEE
17.05.2023
|
| Schlagworte: | |
| ISSN: | 2641-9874 |
| Online-Zugang: | Volltext |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| Abstract | To make machine learning (ML) sustainable and apt to run on the diverse devices where relevant data is, it is essential to compress ML models as needed, while still meeting the required learning quality and time performance. However, how much and when an ML model should be compressed, and where its training should be executed, are hard decisions to make, as they depend on the model itself, the resources of the available nodes, and the data such nodes own. Existing studies focus on each of those aspects individually, however, they do not account for how such decisions can be made jointly and adapted to one another. In this work, we model the network system focusing on the training of DNNs, formalize the above multi-dimensional problem, and, given its NP-hardness, formulate an approximate dynamic programming problem that we solve through the PACT algorithmic framework. Importantly, PACT leverages a time-expanded graph representing the learning process, and a data-driven and theoretical approach for the prediction of the loss evolution to be expected as a consequence of training decisions. We prove that PACT's solutions can get as close to the optimum as desired, at the cost of an increased time complexity, and that, in any case, such complexity is polynomial. Numerical results also show that, even under the most disadvantageous settings, PACT outperforms state-of-the-art alternatives and closely matches the optimal energy cost. |
|---|---|
| AbstractList | To make machine learning (ML) sustainable and apt to run on the diverse devices where relevant data is, it is essential to compress ML models as needed, while still meeting the required learning quality and time performance. However, how much and when an ML model should be compressed, and where its training should be executed, are hard decisions to make, as they depend on the model itself, the resources of the available nodes, and the data such nodes own. Existing studies focus on each of those aspects individually, however, they do not account for how such decisions can be made jointly and adapted to one another. In this work, we model the network system focusing on the training of DNNs, formalize the above multi-dimensional problem, and, given its NP-hardness, formulate an approximate dynamic programming problem that we solve through the PACT algorithmic framework. Importantly, PACT leverages a time-expanded graph representing the learning process, and a data-driven and theoretical approach for the prediction of the loss evolution to be expected as a consequence of training decisions. We prove that PACT's solutions can get as close to the optimum as desired, at the cost of an increased time complexity, and that, in any case, such complexity is polynomial. Numerical results also show that, even under the most disadvantageous settings, PACT outperforms state-of-the-art alternatives and closely matches the optimal energy cost. |
| Author | Levorato, M. Malandrino, F. Karamzade, A. Giacomo, G. Di Chiasserini, C. F. |
| Author_xml | – sequence: 1 givenname: F. surname: Malandrino fullname: Malandrino, F. organization: CNR-IEIIT,Italy – sequence: 2 givenname: G. Di surname: Giacomo fullname: Giacomo, G. Di organization: Politecnico di Torino,Italy – sequence: 3 givenname: A. surname: Karamzade fullname: Karamzade, A. organization: UC Irvine,USA – sequence: 4 givenname: M. surname: Levorato fullname: Levorato, M. organization: UC Irvine,USA – sequence: 5 givenname: C. F. surname: Chiasserini fullname: Chiasserini, C. F. organization: Politecnico di Torino,Italy |
| BookMark | eNo1kM1OwkAURkejiYC8gYvGfXHuTKfTWZIiSgIlMeiW3JneypjSkk7F8Pb-r06-5ORbnCG7aNqGGLsFPgHg5m5RzNf5eqWkkWYiuJAT4EIYrtMzNjbaZFJxKRNIxDkbiDSB2GQ6uWLDEN4455kW6YC9rLB3O9-8RrOiiPJ2f-goBN82ETbl124P1GHvjxRtOvTNt_jh-130RKF97xyFH2-GPUbTI_oara99f7pmlxXWgcZ_HLHn-f0mf4yX64dFPl3GHsD0MVRcQSl45lCSI22VK6XOjLMpgZGCo3YCM5SqBJCpsqlCslZDUhEKa-SI3fz-eiLaHjq_x-60_e8gPwFpTlXg |
| ContentType | Conference Proceeding |
| DBID | 6IE 6IH CBEJK RIE RIO |
| DOI | 10.1109/INFOCOM53939.2023.10229076 |
| DatabaseName | IEEE Electronic Library (IEL) Conference Proceedings IEEE Proceedings Order Plan (POP) 1998-present by volume IEEE Xplore All Conference Proceedings IEEE/IET Electronic Library IEEE Proceedings Order Plans (POP) 1998-present |
| DatabaseTitleList | |
| Database_xml | – sequence: 1 dbid: RIE name: IEEE/IET Electronic Library url: https://ieeexplore.ieee.org/ sourceTypes: Publisher |
| DeliveryMethod | fulltext_linktorsrc |
| Discipline | Engineering |
| EISBN | 9798350334142 |
| EISSN | 2641-9874 |
| EndPage | 10 |
| ExternalDocumentID | 10229076 |
| Genre | orig-research |
| GrantInformation_xml | – fundername: Horizon Europe funderid: 10.13039/100018693 |
| GroupedDBID | 6IE 6IF 6IH 6IK 6IL 6IM 6IN AAJGR AAWTH ABLEC ACGFS ADZIZ ALMA_UNASSIGNED_HOLDINGS BEFXN BFFAM BGNUA BKEBE BPEOZ CBEJK CHZPO IEGSK IJVOP IPLJI M43 OCL RIE RIL RIO |
| ID | FETCH-LOGICAL-i119t-1f051d208ca3ece7b5cd3789cb6e19320a7c2a8a35d11365b65aebb714fea2b93 |
| IEDL.DBID | RIE |
| IngestDate | Wed Aug 27 02:55:18 EDT 2025 |
| IsPeerReviewed | false |
| IsScholarly | true |
| Language | English |
| LinkModel | DirectLink |
| MergedId | FETCHMERGED-LOGICAL-i119t-1f051d208ca3ece7b5cd3789cb6e19320a7c2a8a35d11365b65aebb714fea2b93 |
| PageCount | 10 |
| ParticipantIDs | ieee_primary_10229076 |
| PublicationCentury | 2000 |
| PublicationDate | 2023-May-17 |
| PublicationDateYYYYMMDD | 2023-05-17 |
| PublicationDate_xml | – month: 05 year: 2023 text: 2023-May-17 day: 17 |
| PublicationDecade | 2020 |
| PublicationTitle | Annual Joint Conference of the IEEE Computer and Communications Societies |
| PublicationTitleAbbrev | INFOCOM |
| PublicationYear | 2023 |
| Publisher | IEEE |
| Publisher_xml | – name: IEEE |
| SSID | ssj0008726 |
| Score | 2.29107 |
| Snippet | To make machine learning (ML) sustainable and apt to run on the diverse devices where relevant data is, it is essential to compress ML models as needed, while... |
| SourceID | ieee |
| SourceType | Publisher |
| StartPage | 1 |
| SubjectTerms | Adaptation models Approximation algorithms Costs Energy consumption Heuristic algorithms Prediction algorithms Training |
| Title | Matching DNN Compression and Cooperative Training with Resources and Data Availability |
| URI | https://ieeexplore.ieee.org/document/10229076 |
| hasFullText | 1 |
| inHoldings | 1 |
| isFullTextHit | |
| isPrint | |
| link | http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV1LTwMhECbaeNCLrxrf4eB122VhFzia1kYTu-2hmt4aHtOkidk27baJ_15gt1UPHrwBgYQwGRgGvu9D6EG6MDlVNnGOZFXEmDaRJpZEVLJkKkDyNAtA4Vee52I8lsMarB6wMAAQPp9ByxfDW76dm7VPlbX97cRd5rJ9tM95VoG1dtuu4ElWs4qSWLZf8t6gM-inVAY8SkJb29G_dFTCMdI7_ucETlDzG5CHh7uj5hTtQXGGjn5wCZ6j977bVH06CXfzHHs3r364FlgV1tXnC6hIvvGoVoXAPgeLt_n7VejXVaXCjxs1-6j4uz-b6K33NOo8R7VoQjQjRJYRmTo3s0ksjKJggOvUWMqFNDoDH6zFiptECUVT6-VcUu1MBVpzwqagEi3pBWoU8wIuEbZc0Bi8YpXQTHKtNLMu3OBMKCUypq9Q0y_RZFHxYky2q3P9R_sNOvSG8G_vhN-iRrlcwx06MJtytlreB2t-Ac--oJU |
| linkProvider | IEEE |
| linkToHtml | http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV3LbgIhFCWtbdJ205dN32XR7egwMAMsG63RVEcXtnFneJmYmNHoaNK_LzCjbRdddAcEEsLNhcuFcw4Az9yGybHQkXUkLQJCpAok0ijAnEQTZjiNEw8U7tI0ZaMRH5RgdY-FMcb4z2em5or-LV_P1dqlyurudmIvc8k-OHDSWSVca7fxMholJa8oCnm9k7b6jX4vxtwjUiJc247_paTiD5LW6T-ncAaq35A8ONgdNudgz2QX4OQHm-Al-OjZbdUllGAzTaFz9OKPawZFpm19vjAFzTcclroQ0GVh4TaDv_L9miIX8GUjprOCwfuzCt5br8NGOyhlE4IpQjwP0MQ6mo5CpgQ2ylAZK40p40omxoVroaAqEkzgWDtBl1haYxkpKSITIyLJ8RWoZPPMXAOoKcOhcZpVTBJOpZBE24CDEiYES4i8AVW3RONFwYwx3q7O7R_tT-CoPex1x91O-nYHjp1R3Es8ovegki_X5gEcqk0-XS0fvWW_AFzSo94 |
| openUrl | ctx_ver=Z39.88-2004&ctx_enc=info%3Aofi%2Fenc%3AUTF-8&rfr_id=info%3Asid%2Fsummon.serialssolutions.com&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=proceeding&rft.title=Annual+Joint+Conference+of+the+IEEE+Computer+and+Communications+Societies&rft.atitle=Matching+DNN+Compression+and+Cooperative+Training+with+Resources+and+Data+Availability&rft.au=Malandrino%2C+F.&rft.au=Giacomo%2C+G.+Di&rft.au=Karamzade%2C+A.&rft.au=Levorato%2C+M.&rft.date=2023-05-17&rft.pub=IEEE&rft.eissn=2641-9874&rft.spage=1&rft.epage=10&rft_id=info:doi/10.1109%2FINFOCOM53939.2023.10229076&rft.externalDocID=10229076 |