Convergence of exponentiated gradient algorithms

This paper studies three related algorithms: the (traditional) gradient descent (GD) algorithm, the exponentiated gradient algorithm with positive and negative weights (EG/spl plusmn/ algorithm), and the exponentiated gradient algorithm with unnormalized positive and negative weights (EGU/spl plusmn...

Full description

Saved in:
Bibliographic Details
Published in:IEEE transactions on signal processing Vol. 49; no. 6; pp. 1208 - 1215
Main Authors: Hill, S.I., Williamson, R.C.
Format: Journal Article
Language:English
Published: New York, NY IEEE 01.06.2001
Institute of Electrical and Electronics Engineers
The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
Subjects:
Acoustic applications
Algorithm design and analysis
Algorithms
Applied sciences
Communities
Convergence
Detection, estimation, filtering, equalization, prediction
Echo cancellers
Exact sciences and technology
Information, signal and communications theory
Learning
Learning theory
Least squares approximation
Mathematical analysis
Mean square error methods
Performance analysis
Side effects
Signal analysis
Signal and communications theory
Signal processing
Signal processing algorithms
Signal, noise
Steady-state
Telecommunications and information theory
Algorithm performance
Acoustic signal
Signal processing
Acoustic signal processing
Room acoustics
Echo suppression
Learning algorithm
Gradient method
Mean square error
Convergence speed
ISSN:1053-587X, 1941-0476
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:This paper studies three related algorithms: the (traditional) gradient descent (GD) algorithm, the exponentiated gradient algorithm with positive and negative weights (EG/spl plusmn/ algorithm), and the exponentiated gradient algorithm with unnormalized positive and negative weights (EGU/spl plusmn/ algorithm). These algorithms have been previously analyzed using the "mistake-bound framework" in the computational learning theory community. We perform a traditional signal processing analysis in terms of the mean square error. A relationship between the learning rate and the mean squared error (MSE) of predictions is found for the family of algorithms. This is used to compare the performance of the algorithms by choosing learning rates such that they converge to the same steady-state MSE. We demonstrate that if the target weight vector is sparse, the EG/spl plusmn/ algorithm typically converges more quickly than the GD or EGU/spl plusmn/ algorithms that perform very similarly. A side effect of our analysis is a reparametrization of the algorithms that provides insights into their behavior. The general form of the results we obtain are consistent with those obtained in the mistake-bound framework. The application of the algorithms to acoustic echo cancellation is then studied, and it is shown in some circumstances that the EG/spl plusmn/ algorithm will converge faster than the other two algorithms.
Bibliography:ObjectType-Article-2
SourceType-Scholarly Journals-1
ObjectType-Feature-1
content type line 14
content type line 23
ISSN:1053-587X
1941-0476
DOI:10.1109/78.923303