M22: A Communication-Efficient Algorithm for Federated Learning Inspired by Rate-Distortion

In federated learning (FL), the communication constraint between the remote clients and the Parameter Server (PS) is a crucial bottleneck. For this reason, model updates must be compressed so as to minimize the loss in accuracy resulting from the communication constraint. This paper proposes "M...

Full description

Saved in:
Bibliographic Details
Published in:IEEE transactions on communications Vol. 72; no. 2; pp. 845 - 860
Main Authors: Liu, Yangyi, Rini, Stefano, Salehkalaibar, Sadaf, Chen, Jun
Format: Journal Article
Language:English
Published: New York IEEE 01.02.2024
The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
Subjects:
Accuracy
Algorithms
Artificial neural networks
Codes
Communication
Compression algorithms
Degrees of freedom
Distortion
Distortion measurement
DNN gradient modeling
Federated learning
gradient compression
gradient quantization
gradient sparsification
Iterative methods
Machine learning
Mathematical models
Parameters
Quantization (signal)
Rate-distortion
Reconstruction
Training
ISSN:0090-6778, 1558-0857
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:In federated learning (FL), the communication constraint between the remote clients and the Parameter Server (PS) is a crucial bottleneck. For this reason, model updates must be compressed so as to minimize the loss in accuracy resulting from the communication constraint. This paper proposes "M-magnitude weighted L2 distortion + 2 degrees of freedom" (M22) algorithm, a rate-distortion inspired approach to gradient compression for federated training of deep neural networks (DNNs). In particular, we propose a family of distortion measures between the original gradient and the reconstruction we referred to as "<inline-formula> <tex-math notation="LaTeX">M </tex-math></inline-formula>-magnitude weighted <inline-formula> <tex-math notation="LaTeX">L_{2} </tex-math></inline-formula>" distortion, and we assume that gradient updates follow an i.i.d. distribution - generalized normal or Weibull, which have two degrees of freedom. In both the distortion measure and the gradient distribution, there is one free parameter for each that can be fitted as a function of the iteration number. Given a choice of gradient distribution and distortion measure, we design the quantizer to minimize the expected distortion in gradient reconstruction. To measure the gradient compression performance under a communication constraint, we define the per-bit accuracy as the optimal improvement in accuracy that one bit of communication brings to the centralized model over the training period. Using this performance measure, we systematically benchmark the choice of gradient distribution and distortion measure. We provide substantial insights on the role of these choices and argue that significant performance improvements can be attained using such a rate-distortion inspired compressor.
Bibliography:ObjectType-Article-1
SourceType-Scholarly Journals-1
ObjectType-Feature-2
content type line 14
ISSN:0090-6778
1558-0857
DOI:10.1109/TCOMM.2023.3327778