The Kernel Conjugate Gradient Algorithms

Kernel methods have been successfully applied to nonlinear problems in machine learning and signal processing. Various kernel-based algorithms have been proposed over the last two decades. In this paper, we investigate the kernel conjugate gradient (KCG) algorithms in both batch and online modes. By...

Full description

Saved in:
Bibliographic Details
Published in:IEEE transactions on signal processing Vol. 66; no. 16; pp. 4377 - 4387
Main Authors: Zhang, Ming, Wang, Xiaojian, Chen, Xiaoming, Zhang, Anxue
Format: Journal Article
Language:English
Published: New York IEEE 15.08.2018
The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
Subjects:
Adaptive algorithms
Algorithms
Computational efficiency
conjugate gradient algorithm
Conjugates
Convergence
Criteria
Hilbert space
Iterative methods
Kernel
Machine learning
Machine learning algorithms
nonlinear processing
online sparsification
Optimization
regularization
Reproducing kernel Hilbert space
Robustness (mathematics)
Signal processing
Signal processing algorithms
ISSN:1053-587X, 1941-0476
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:Kernel methods have been successfully applied to nonlinear problems in machine learning and signal processing. Various kernel-based algorithms have been proposed over the last two decades. In this paper, we investigate the kernel conjugate gradient (KCG) algorithms in both batch and online modes. By expressing the solution vector of CG algorithm as a linear combination of the input vectors and using the kernel trick, we developed the KCG algorithm for batch mode. Because the CG algorithm is iterative in nature, it can greatly reduce the computations by the technique of reduced-rank processing. Moreover, the reduced-rank processing can provide the robustness against the problem of overlearning. The online KCG algorithm is also derived, which converges as fast as the kernel recursive least squares (KRLS) algorithm, but the computational cost is only a quarter of that of the KRLS algorithm. Another attractive feature of the online KCG algorithm compared with other kernel adaptive algorithms is that it does not require the user-defined parameters. To control the growth of data size in online applications, a simple sparsification criterion based on the angles among elements in reproducing kernel Hilbert space is proposed. The angle criterion is equivalent to the coherence criterion but does not require the kernel to be unit norm. Finally, numerical experiments are provided to illustrate the effectiveness of the proposed algorithms.
Bibliography:ObjectType-Article-1
SourceType-Scholarly Journals-1
ObjectType-Feature-2
content type line 14
ISSN:1053-587X
1941-0476
DOI:10.1109/TSP.2018.2853109