Learning Dynamics of Low-Precision Clipped SGD with Momentum

In this work, we present and study a low-precision variant of the stochastic gradient descent (SGD) algorithm with adaptive quantization. In particular, fixed-rate probabilistic uniform quantizers with varying quantization steps and mid-values are used to compress the parameter vectors. Gradient cli...

Ausführliche Beschreibung

Gespeichert in:
Bibliographische Detailangaben
Veröffentlicht in:Proceedings of the ... IEEE International Conference on Acoustics, Speech and Signal Processing (1998) S. 6075 - 6079
Hauptverfasser: Nassif, Roula, Kar, Soummya, Vlaski, Stefan
Format: Tagungsbericht
Sprache:Englisch
Veröffentlicht: IEEE 14.04.2024
Schlagworte:
Acoustics
differential randomized quantization
Fixed rate
gradient clipping
Heuristic algorithms
momentum
online learning
Probabilistic logic
Quantization (signal)
Signal processing algorithms
Simulation
Vectors
ISSN:2379-190X
Online-Zugang:Volltext
Tags: Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
Beschreibung
Zusammenfassung:In this work, we present and study a low-precision variant of the stochastic gradient descent (SGD) algorithm with adaptive quantization. In particular, fixed-rate probabilistic uniform quantizers with varying quantization steps and mid-values are used to compress the parameter vectors. Gradient clipping and momentum are used to guarantee that the quantizer inputs fall within the representable region of the fixed-rate quantizer and to reduce the impact of the stochastic gradient noise, respectively. We show that, despite the low-precision representation, the quantized variant of the clipped SGD algorithm with momentum is able to converge in the mean-square-error sense. Simulation results illustrate the theoretical findings and the effectiveness of the proposed approach.
ISSN:2379-190X
DOI:10.1109/ICASSP48485.2024.10447855