Optimal stochastic gradient descent algorithm for filtering

Stochastic Gradient Descent (SGD) is a fundamental optimization technique in machine learning, due to its efficiency in handling large-scale data. Unlike typical SGD applications, which rely on stochastic approximations, this work explores the convergence properties of SGD from a deterministic persp...

Full description

Saved in:
Bibliographic Details
Published in:Digital signal processing Vol. 155; p. 104731
Main Authors: Turali, M. Yigit, Koc, Ali T., Kozat, Suleyman S.
Format: Journal Article
Language:English
Published: Elsevier Inc 01.12.2024
Subjects:
Learning rate
Linear filtering
Optimization
Stochastic gradient descent
Linear filtering
Stochastic gradient descent
Learning rate
Optimization
ISSN:1051-2004
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:Stochastic Gradient Descent (SGD) is a fundamental optimization technique in machine learning, due to its efficiency in handling large-scale data. Unlike typical SGD applications, which rely on stochastic approximations, this work explores the convergence properties of SGD from a deterministic perspective. We address the crucial aspect of learning rate settings, a common obstacle in optimizing SGD performance, particularly in complex environments. In contrast to traditional methods that often provide convergence results based on statistical expectations (which are usually not justified), our approach introduces universally applicable learning rates. These rates ensure that a model trained with SGD matches the performance of the best linear filter asymptotically, applicable irrespective of the data sequence length and independent of statistical assumptions about the data. By establishing learning rates that scale as μ=O(1t), we offer a solution that sidesteps the need for prior data knowledge, a prevalent limitation in real-world applications. To this end, we provide a robust framework for SGD's application across varied settings, guaranteeing convergence results that hold under both deterministic and stochastic scenarios without any underlying assumptions.
ISSN:1051-2004
DOI:10.1016/j.dsp.2024.104731