Learning Dynamics of Low-Precision Clipped SGD with Momentum

In this work, we present and study a low-precision variant of the stochastic gradient descent (SGD) algorithm with adaptive quantization. In particular, fixed-rate probabilistic uniform quantizers with varying quantization steps and mid-values are used to compress the parameter vectors. Gradient cli...

Full description

Saved in:
Bibliographic Details
Published in:Proceedings of the ... IEEE International Conference on Acoustics, Speech and Signal Processing (1998) pp. 6075 - 6079
Main Authors: Nassif, Roula, Kar, Soummya, Vlaski, Stefan
Format: Conference Proceeding
Language:English
Published: IEEE 14.04.2024
Subjects:
Acoustics
differential randomized quantization
Fixed rate
gradient clipping
Heuristic algorithms
momentum
online learning
Probabilistic logic
Quantization (signal)
Signal processing algorithms
Simulation
Vectors
ISSN:2379-190X
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:In this work, we present and study a low-precision variant of the stochastic gradient descent (SGD) algorithm with adaptive quantization. In particular, fixed-rate probabilistic uniform quantizers with varying quantization steps and mid-values are used to compress the parameter vectors. Gradient clipping and momentum are used to guarantee that the quantizer inputs fall within the representable region of the fixed-rate quantizer and to reduce the impact of the stochastic gradient noise, respectively. We show that, despite the low-precision representation, the quantized variant of the clipped SGD algorithm with momentum is able to converge in the mean-square-error sense. Simulation results illustrate the theoretical findings and the effectiveness of the proposed approach.
ISSN:2379-190X
DOI:10.1109/ICASSP48485.2024.10447855