Sum-Rate-Distortion Function for Indirect Multiterminal Source Coding in Federated Learning

One of the main focus in federated learning (FL) is the communication efficiency since a large number of participating edge devices send their updates to the edge server at each round of the model training. Existing works reconstruct each model update from edge devices and implicitly assume that the...

Full description

Saved in:
Bibliographic Details
Published in:2021 IEEE International Symposium on Information Theory (ISIT) pp. 2161 - 2166
Main Authors: Zhang, Naifu, Tao, Meixia, Wang, Jia
Format: Conference Proceeding
Language:English
Published: IEEE 12.07.2021
Subjects:
Collaborative work
Estimation
Rate-distortion
Redundancy
Source coding
Training
Upper bound
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Abstract One of the main focus in federated learning (FL) is the communication efficiency since a large number of participating edge devices send their updates to the edge server at each round of the model training. Existing works reconstruct each model update from edge devices and implicitly assume that the local model updates are independent over edge devices. In FL, however, the model update is an indirect multi-terminal source coding problem, also called as the CEO problem where each edge device cannot observe directly the gradient that is to be reconstructed at the decoder, but is rather provided only with a noisy version. The existing works do not leverage the redundancy in the information transmitted by different edges. This paper studies the rate region for the indirect multiterminal source coding problem in FL. The goal is to obtain the minimum achievable rate at a particular upper bound of gradient variance. We obtain the rate region for the quadratic vector Gaussian CEO problem under unbiased estimator and derive an explicit formula of the sum-rate-distortion function in the special case where gradient are identical over edge device and dimension. Finally, we analyse communication efficiency of convex Mini-batched SGD and non-convex Minibatched SGD based on the sum-rate-distortion function, respectively.
AbstractList One of the main focus in federated learning (FL) is the communication efficiency since a large number of participating edge devices send their updates to the edge server at each round of the model training. Existing works reconstruct each model update from edge devices and implicitly assume that the local model updates are independent over edge devices. In FL, however, the model update is an indirect multi-terminal source coding problem, also called as the CEO problem where each edge device cannot observe directly the gradient that is to be reconstructed at the decoder, but is rather provided only with a noisy version. The existing works do not leverage the redundancy in the information transmitted by different edges. This paper studies the rate region for the indirect multiterminal source coding problem in FL. The goal is to obtain the minimum achievable rate at a particular upper bound of gradient variance. We obtain the rate region for the quadratic vector Gaussian CEO problem under unbiased estimator and derive an explicit formula of the sum-rate-distortion function in the special case where gradient are identical over edge device and dimension. Finally, we analyse communication efficiency of convex Mini-batched SGD and non-convex Minibatched SGD based on the sum-rate-distortion function, respectively.
Author Zhang, Naifu
Tao, Meixia
Wang, Jia
Author_xml – sequence: 1
  givenname: Naifu
  surname: Zhang
  fullname: Zhang, Naifu
  email: arthaslery@sjtu.edu.cn
  organization: Shanghai Jiao Tong University,Department of Electronic Engineering,Shanghai,P. R. China,200240
– sequence: 2
  givenname: Meixia
  surname: Tao
  fullname: Tao, Meixia
  email: mxtao@sjtu.edu.cn
  organization: Shanghai Jiao Tong University,Department of Electronic Engineering,Shanghai,P. R. China,200240
– sequence: 3
  givenname: Jia
  surname: Wang
  fullname: Wang, Jia
  email: jiawang@sjtu.edu.cn
  organization: Shanghai Jiao Tong University,Department of Electronic Engineering,Shanghai,P. R. China,200240
BookMark eNotj81KxDAYRSPoQsd5AkHyAq35nSRLqY4WKoIdVy6GNPkqgTaRTLrw7R2cWd3LhXPg3qDLmCIgdE9JTSkxD23f7oSkStSMMFobSTVR4gKtjdJUcr3RjBh9jb76Za4-bIHqKRxKyiWkiLdLdP9lTBm30YcMruC3ZSqhQJ5DtBPu05Id4Cb5EL9xOELgIR9FHndgczyut-hqtNMB1udcoc_t8655rbr3l7Z57KrACC8V82QjxoFw580onVSWGCpHqrgfhOfScwGCaaqdFcCIEEoCjNyzgWlPlOcrdHfyBgDY_-Qw2_y7P1_mf4cfUaM
ContentType Conference Proceeding
DBID 6IE
6IH
CBEJK
RIE
RIO
DOI 10.1109/ISIT45174.2021.9518074
DatabaseName IEEE Electronic Library (IEL) Conference Proceedings
IEEE Proceedings Order Plan (POP) 1998-present by volume
IEEE Xplore All Conference Proceedings
IEEE Electronic Library (IEL)
IEEE Proceedings Order Plans (POP) 1998-present
DatabaseTitleList
Database_xml – sequence: 1
  dbid: RIE
  name: IEEE/IET Electronic Library (IEL) (UW System Shared)
  url: https://ieeexplore.ieee.org/
  sourceTypes: Publisher
DeliveryMethod fulltext_linktorsrc
EISBN 9781538682098
1538682095
EndPage 2166
ExternalDocumentID 9518074
Genre orig-research
GrantInformation_xml – fundername: NSF of China
  grantid: 61941106,61771305
  funderid: 10.13039/501100001809
GroupedDBID 6IE
6IH
CBEJK
RIE
RIO
ID FETCH-LOGICAL-i203t-2d064fb03cd9f5c57a0915f173db4d35d34e42818ca4e204475eef3d2b28d07d3
IEDL.DBID RIE
ISICitedReferencesCount 4
ISICitedReferencesURI http://www.webofscience.com/api/gateway?GWVersion=2&SrcApp=Summon&SrcAuth=ProQuest&DestLinkType=CitingArticles&DestApp=WOS_CPL&KeyUT=000701502202043&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D
IngestDate Thu Jun 29 18:37:48 EDT 2023
IsPeerReviewed false
IsScholarly false
Language English
LinkModel DirectLink
MergedId FETCHMERGED-LOGICAL-i203t-2d064fb03cd9f5c57a0915f173db4d35d34e42818ca4e204475eef3d2b28d07d3
PageCount 6
ParticipantIDs ieee_primary_9518074
PublicationCentury 2000
PublicationDate 2021-July-12
PublicationDateYYYYMMDD 2021-07-12
PublicationDate_xml – month: 07
  year: 2021
  text: 2021-July-12
  day: 12
PublicationDecade 2020
PublicationTitle 2021 IEEE International Symposium on Information Theory (ISIT)
PublicationTitleAbbrev ISIT
PublicationYear 2021
Publisher IEEE
Publisher_xml – name: IEEE
Score 1.7985958
Snippet One of the main focus in federated learning (FL) is the communication efficiency since a large number of participating edge devices send their updates to the...
SourceID ieee
SourceType Publisher
StartPage 2161
SubjectTerms Collaborative work
Estimation
Rate-distortion
Redundancy
Source coding
Training
Upper bound
Title Sum-Rate-Distortion Function for Indirect Multiterminal Source Coding in Federated Learning
URI https://ieeexplore.ieee.org/document/9518074
WOSCitedRecordID wos000701502202043&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D
hasFullText 1
inHoldings 1
isFullTextHit
isPrint
link http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV1NSwMxEB3a4sGTSit-k4NH024-ttmeq8VeSrEVCh5KPiZS0K3U1t9vkl0qghdvYckmMCH73k7y3gDcaiHiYZChUnNJZe502HM8p14VAWED3heFTcUm1GRSLBaDaQPu9loYREyXz7Abm-ks363tLqbKeoENRO-WJjSV6ldarVr0y7JBbzwbz2U0Xg5_fZx1686_qqYk0Bgd_W-6Y-j8qO_IdI8rJ9DAsg0vs907fQrEkN4nY48YTzIKoJQagXmScVnBE6k0tdUllzcyS9l5MlzHwcgqvBTtI8JAjtTeqq8deB49zIePtC6MQFc8E1vKXSAS3mTCuoHPba50QP3cMyWckU7kTkiU0ebJaok8i55-iF44bnjhMuXEKbTKdYlnQIRjWlnNkDMv-0YbDAQGPQ_fwYwbpc-hHQOz_Ki8L5Z1TC7-fnwJhzH2MffJ-BW0tpsdXsOB_dquPjc3acG-AU_hmQQ
linkProvider IEEE
linkToHtml http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV3LSgMxFL3UKuhKpRXfZuHStJk8zHRdLR2spdgKBRclr5GCTqW2fr9JZqgIbtyFMJOBGzLn5CbnXIBrxVg4DNKYK8oxF1b5NUcFzmXqEdbjfZqaWGxCDofpdNoZ1eBmo4VxzsXLZ64VmvEs3y7MOqTK2p4NBO-WLdgWnFNSqrUq2W9COu1snE14sF72-z6atKrHf9VNibDR2__fBw-g-aO_Q6MNshxCzRUNeBmv3_GTp4b4Llp7hIiinoel2PDcE2VFCVCoVNWW11ze0Djm51F3EQZDc_9SMJDwA1lUuau-NuG5dz_p9nFVGgHPKWErTK2nErkmzNhOLoyQyuO-yBPJrOaWCcu448HoySjuKAmufs7lzFJNU0ukZUdQLxaFOwbEbKKkUYmjSc5vtdLOUxiXU_8nJFRLdQKNEJjZR-l-Maticvp39xXs9iePg9kgGz6cwV6Yh5AJTeg51FfLtbuAHfO1mn8uL-PkfQNyLJxL
openUrl ctx_ver=Z39.88-2004&ctx_enc=info%3Aofi%2Fenc%3AUTF-8&rfr_id=info%3Asid%2Fsummon.serialssolutions.com&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=proceeding&rft.title=2021+IEEE+International+Symposium+on+Information+Theory+%28ISIT%29&rft.atitle=Sum-Rate-Distortion+Function+for+Indirect+Multiterminal+Source+Coding+in+Federated+Learning&rft.au=Zhang%2C+Naifu&rft.au=Tao%2C+Meixia&rft.au=Wang%2C+Jia&rft.date=2021-07-12&rft.pub=IEEE&rft.spage=2161&rft.epage=2166&rft_id=info:doi/10.1109%2FISIT45174.2021.9518074&rft.externalDocID=9518074