Learning Dynamics of Low-Precision Clipped SGD with Momentum

In this work, we present and study a low-precision variant of the stochastic gradient descent (SGD) algorithm with adaptive quantization. In particular, fixed-rate probabilistic uniform quantizers with varying quantization steps and mid-values are used to compress the parameter vectors. Gradient cli...

Celý popis

Uloženo v:
Podrobná bibliografie
Vydáno v:Proceedings of the ... IEEE International Conference on Acoustics, Speech and Signal Processing (1998) s. 6075 - 6079
Hlavní autoři: Nassif, Roula, Kar, Soummya, Vlaski, Stefan
Médium: Konferenční příspěvek
Jazyk:angličtina
Vydáno: IEEE 14.04.2024
Témata:
Acoustics
differential randomized quantization
Fixed rate
gradient clipping
Heuristic algorithms
momentum
online learning
Probabilistic logic
Quantization (signal)
Signal processing algorithms
Simulation
Vectors
ISSN:2379-190X
On-line přístup:Získat plný text
Tagy: Přidat tag
Žádné tagy, Buďte první, kdo vytvoří štítek k tomuto záznamu!
Popis
Shrnutí:In this work, we present and study a low-precision variant of the stochastic gradient descent (SGD) algorithm with adaptive quantization. In particular, fixed-rate probabilistic uniform quantizers with varying quantization steps and mid-values are used to compress the parameter vectors. Gradient clipping and momentum are used to guarantee that the quantizer inputs fall within the representable region of the fixed-rate quantizer and to reduce the impact of the stochastic gradient noise, respectively. We show that, despite the low-precision representation, the quantized variant of the clipped SGD algorithm with momentum is able to converge in the mean-square-error sense. Simulation results illustrate the theoretical findings and the effectiveness of the proposed approach.
ISSN:2379-190X
DOI:10.1109/ICASSP48485.2024.10447855