The Kernel Conjugate Gradient Algorithms

Kernel methods have been successfully applied to nonlinear problems in machine learning and signal processing. Various kernel-based algorithms have been proposed over the last two decades. In this paper, we investigate the kernel conjugate gradient (KCG) algorithms in both batch and online modes. By...

Full description

Saved in:

Bibliographic Details
Published in:	IEEE transactions on signal processing Vol. 66; no. 16; pp. 4377 - 4387
Main Authors:	Zhang, Ming, Wang, Xiaojian, Chen, Xiaoming, Zhang, Anxue
Format:	Journal Article
Language:	English
Published:	New York IEEE 15.08.2018 The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
Subjects:	Adaptive algorithms Algorithms Computational efficiency conjugate gradient algorithm Conjugates Convergence Criteria Hilbert space Iterative methods Kernel Machine learning Machine learning algorithms nonlinear processing online sparsification Optimization regularization Reproducing kernel Hilbert space Robustness (mathematics) Signal processing Signal processing algorithms
ISSN:	1053-587X, 1941-0476
Online Access:	Get full text
Tags:	Add Tag No Tags, Be the first to tag this record!

Abstract	Kernel methods have been successfully applied to nonlinear problems in machine learning and signal processing. Various kernel-based algorithms have been proposed over the last two decades. In this paper, we investigate the kernel conjugate gradient (KCG) algorithms in both batch and online modes. By expressing the solution vector of CG algorithm as a linear combination of the input vectors and using the kernel trick, we developed the KCG algorithm for batch mode. Because the CG algorithm is iterative in nature, it can greatly reduce the computations by the technique of reduced-rank processing. Moreover, the reduced-rank processing can provide the robustness against the problem of overlearning. The online KCG algorithm is also derived, which converges as fast as the kernel recursive least squares (KRLS) algorithm, but the computational cost is only a quarter of that of the KRLS algorithm. Another attractive feature of the online KCG algorithm compared with other kernel adaptive algorithms is that it does not require the user-defined parameters. To control the growth of data size in online applications, a simple sparsification criterion based on the angles among elements in reproducing kernel Hilbert space is proposed. The angle criterion is equivalent to the coherence criterion but does not require the kernel to be unit norm. Finally, numerical experiments are provided to illustrate the effectiveness of the proposed algorithms.
AbstractList	Kernel methods have been successfully applied to nonlinear problems in machine learning and signal processing. Various kernel-based algorithms have been proposed over the last two decades. In this paper, we investigate the kernel conjugate gradient (KCG) algorithms in both batch and online modes. By expressing the solution vector of CG algorithm as a linear combination of the input vectors and using the kernel trick, we developed the KCG algorithm for batch mode. Because the CG algorithm is iterative in nature, it can greatly reduce the computations by the technique of reduced-rank processing. Moreover, the reduced-rank processing can provide the robustness against the problem of overlearning. The online KCG algorithm is also derived, which converges as fast as the kernel recursive least squares (KRLS) algorithm, but the computational cost is only a quarter of that of the KRLS algorithm. Another attractive feature of the online KCG algorithm compared with other kernel adaptive algorithms is that it does not require the user-defined parameters. To control the growth of data size in online applications, a simple sparsification criterion based on the angles among elements in reproducing kernel Hilbert space is proposed. The angle criterion is equivalent to the coherence criterion but does not require the kernel to be unit norm. Finally, numerical experiments are provided to illustrate the effectiveness of the proposed algorithms.
Author	Zhang, Anxue Wang, Xiaojian Zhang, Ming Chen, Xiaoming
Author_xml	– sequence: 1 givenname: Ming orcidid: 0000-0002-5796-4754 surname: Zhang fullname: Zhang, Ming email: itjerry@163.com organization: School of Electronic and Information Engineering, Xi'an Jiaotong University, Xi'an, China – sequence: 2 givenname: Xiaojian surname: Wang fullname: Wang, Xiaojian email: wangxj235@163.com organization: School of Electronic and Information Engineering, Xi'an Jiaotong University, Xi'an, China – sequence: 3 givenname: Xiaoming orcidid: 0000-0003-3151-1690 surname: Chen fullname: Chen, Xiaoming email: xiaoming.chen@mail.xjtu.edu.cn organization: School of Electronic and Information Engineering, Xi'an Jiaotong University, Xi'an, China – sequence: 4 givenname: Anxue orcidid: 0000-0002-4690-2718 surname: Zhang fullname: Zhang, Anxue email: anxuezhang@mail.xjtu.edu.cn organization: School of Electronic and Information Engineering, Xi'an Jiaotong University, Xi'an, China
BookMark	eNp9kEFLw0AQRhepYFu9C14CXrykziS72d1jKVrFgoIVvC3bzaRNSZO62Rz896a0ePDgaT6G783AG7FB3dTE2DXCBBH0_fL9bZIAqkmiRNovztgQNccYuMwGfQaRxkLJzws2atstAHKusyG7W24oeiFfUxXNmnrbrW2gaO5tXlIdomm1bnwZNrv2kp0Xtmrp6jTH7OPxYTl7ihev8-fZdBG7RGOIV1w45XKeQFpgobQUK-04WEICYYtCE1lIpUoyTGQO2pHIFUeVpytbOEnpmN0e7-5989VRG8y26XzdvzQJogShtRB9Kzu2nG_a1lNhXBlsKJs6eFtWBsEcrJjeijlYMScrPQh_wL0vd9Z__4fcHJGSiH7rioOSUqY_cjxt4g
CODEN	ITPRED
CitedBy_id	crossref_primary_10_1016_j_patcog_2023_110198 crossref_primary_10_1109_LSP_2019_2907480 crossref_primary_10_3390_math12243935 crossref_primary_10_3390_e21080785 crossref_primary_10_1109_ACCESS_2020_2968753 crossref_primary_10_3389_fpls_2022_849606 crossref_primary_10_1016_j_sigpro_2020_107836 crossref_primary_10_1016_j_sigpro_2024_109837 crossref_primary_10_1109_TCSII_2023_3303365 crossref_primary_10_1016_j_sigpro_2022_108787 crossref_primary_10_1109_TCSII_2023_3304698 crossref_primary_10_1007_s11277_024_11084_5 crossref_primary_10_1049_ell2_12809 crossref_primary_10_1109_LSP_2021_3081381 crossref_primary_10_1109_TCYB_2019_2959834 crossref_primary_10_1109_TSP_2021_3065173 crossref_primary_10_1016_j_neunet_2023_04_040 crossref_primary_10_1016_j_neucom_2025_130234 crossref_primary_10_1007_s00034_021_01804_8 crossref_primary_10_1109_TCSII_2019_2957781 crossref_primary_10_1109_TAES_2025_3540050 crossref_primary_10_1016_j_dsp_2025_104997 crossref_primary_10_3390_electronics11182819 crossref_primary_10_1007_s11277_021_08155_2 crossref_primary_10_1109_TIM_2022_3173628 crossref_primary_10_1016_j_jfranklin_2022_04_002 crossref_primary_10_1109_LSP_2022_3167897 crossref_primary_10_3390_sym11101323 crossref_primary_10_1007_s00034_024_02654_w crossref_primary_10_1109_TFUZZ_2023_3348847 crossref_primary_10_1109_LSP_2023_3284842 crossref_primary_10_1007_s00034_022_02258_2
Cites_doi	10.1049/iet-spr.2011.0324 10.1109/DSP-SPE.2013.6642587 10.1109/ICASSP.2006.1661394 10.1109/TSP.2007.907881 10.1109/18.737524 10.1155/2008/784292 10.1007/978-3-319-18842-3 10.1090/S0002-9947-1950-0051437-7 10.1016/0022-247X(71)90184-3 10.1109/TSP.2004.830991 10.6028/jres.049.044 10.1109/IJCNN.2005.1556088 10.1137/1.9780898719697 10.1109/ICASSP.2017.7952666 10.1109/TSP.2009.2022007 10.1162/089976698300017467 10.1109/TNNLS.2012.2200500 10.1109/TSP.2009.2022857 10.1109/TSP.2008.2009895 10.1109/TNNLS.2013.2285783 10.1109/TSP.2015.2437835 10.1002/cem.728 10.1109/TSP.2010.2096420 10.1109/TSP.2004.830985 10.1145/355984.355989 10.1007/978-1-4757-3264-1 10.56021/9781421407944 10.1109/TNNLS.2011.2178446 10.1109/TSP.2015.2457396 10.1109/TNNLS.2013.2258936 10.1109/TSP.2015.2493990 10.1109/TSP.2016.2605075 10.1109/TSP.2017.2726991
ContentType	Journal Article
Copyright	Copyright The Institute of Electrical and Electronics Engineers, Inc. (IEEE) 2018
Copyright_xml	– notice: Copyright The Institute of Electrical and Electronics Engineers, Inc. (IEEE) 2018
DBID	97E RIA RIE AAYXX CITATION 7SC 7SP 8FD JQ2 L7M L~C L~D
DOI	10.1109/TSP.2018.2853109
DatabaseName	IEEE All-Society Periodicals Package (ASPP) 2005–Present IEEE All-Society Periodicals Package (ASPP) 1998–Present IEEE Electronic Library (IEL) CrossRef Computer and Information Systems Abstracts Electronics & Communications Abstracts Technology Research Database ProQuest Computer Science Collection Advanced Technologies Database with Aerospace Computer and Information Systems Abstracts Academic Computer and Information Systems Abstracts Professional
DatabaseTitle	CrossRef Technology Research Database Computer and Information Systems Abstracts – Academic Electronics & Communications Abstracts ProQuest Computer Science Collection Computer and Information Systems Abstracts Advanced Technologies Database with Aerospace Computer and Information Systems Abstracts Professional
DatabaseTitleList	Technology Research Database
Database_xml	– sequence: 1 dbid: RIE name: IEEE/IET Electronic Library (IEL) (UW System Shared) url: https://ieeexplore.ieee.org/ sourceTypes: Publisher
DeliveryMethod	fulltext_linktorsrc
Discipline	Engineering
EISSN	1941-0476
EndPage	4387
ExternalDocumentID	10_1109_TSP_2018_2853109 8408777
Genre	orig-research
GrantInformation_xml	– fundername: National Natural Science Foundation of China grantid: 61471292 funderid: 10.13039/501100001809
GroupedDBID	-~X .DC 0R~ 29I 4.4 5GY 6IK 85S 97E AAJGR AARMG AASAJ AAWTH ABAZT ABQJQ ABVLG ACGFO ACIWK ACNCT AENEX AGQYO AHBIQ AJQPL AKJIK AKQYR ALMA_UNASSIGNED_HOLDINGS ASUFR ATWAV BEFXN BFFAM BGNUA BKEBE BPEOZ CS3 EBS EJD F5P HZ~ IFIPE IPLJI JAVBF LAI MS~ O9- OCL P2P RIA RIE RNS TAE TN5 AAYXX CITATION 7SC 7SP 8FD JQ2 L7M L~C L~D RIG
ID	FETCH-LOGICAL-c291t-b45c8cd4203f1f8975b9c40ae1e05aff9eea037826127d09ce5d8418d3bafc7e3
IEDL.DBID	RIE
ISICitedReferencesCount	40
ISICitedReferencesURI	http://www.webofscience.com/api/gateway?GWVersion=2&SrcApp=Summon&SrcAuth=ProQuest&DestLinkType=CitingArticles&DestApp=WOS_CPL&KeyUT=000439360200001&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D
ISSN	1053-587X
IngestDate	Mon Jun 30 10:19:14 EDT 2025 Tue Nov 18 21:09:03 EST 2025 Sat Nov 29 04:10:46 EST 2025 Wed Aug 27 02:39:32 EDT 2025
IsPeerReviewed	true
IsScholarly	true
Issue	16
Language	English
License	https://ieeexplore.ieee.org/Xplorehelp/downloads/license-information/IEEE.html https://doi.org/10.15223/policy-029 https://doi.org/10.15223/policy-037
LinkModel	DirectLink
MergedId	FETCHMERGED-LOGICAL-c291t-b45c8cd4203f1f8975b9c40ae1e05aff9eea037826127d09ce5d8418d3bafc7e3
Notes	ObjectType-Article-1 SourceType-Scholarly Journals-1 ObjectType-Feature-2 content type line 14
ORCID	0000-0003-3151-1690 0000-0002-5796-4754 0000-0002-4690-2718
PQID	2117059955
PQPubID	85478
PageCount	11
ParticipantIDs	crossref_citationtrail_10_1109_TSP_2018_2853109 proquest_journals_2117059955 crossref_primary_10_1109_TSP_2018_2853109 ieee_primary_8408777
PublicationCentury	2000
PublicationDate	2018-08-15
PublicationDateYYYYMMDD	2018-08-15
PublicationDate_xml	– month: 08 year: 2018 text: 2018-08-15 day: 15
PublicationDecade	2010
PublicationPlace	New York
PublicationPlace_xml	– name: New York
PublicationTitle	IEEE transactions on signal processing
PublicationTitleAbbrev	TSP
PublicationYear	2018
Publisher	IEEE The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
Publisher_xml	– name: IEEE – name: The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
References	ref13 ref34 ref12 blanchard (ref39) 2010; 23 ref15 luenberger (ref20) 2016 ref36 ref14 bach (ref6) 2002; 3 ref31 ref33 ref11 ref10 haykin (ref19) 2014 ref1 ref17 ratliff (ref24) 2007 ref38 mcdonald (ref32) 2013 ref16 golub (ref35) 2013 fletcher (ref23) 1987 schölkopf (ref2) 2002 kreyszig (ref30) 1978 ref25 ref42 ref41 ref22 ref44 ref21 ref43 ref28 ref27 ref29 ref8 ref7 hiemstra (ref37) 2003 ref9 ref4 widrow (ref18) 1985 ref3 rosipal (ref26) 2001; 2 ref5 ref40
References_xml	– ident: ref21 doi: 10.1049/iet-spr.2011.0324 – ident: ref43 doi: 10.1109/DSP-SPE.2013.6642587 – ident: ref11 doi: 10.1109/ICASSP.2006.1661394 – ident: ref12 doi: 10.1109/TSP.2007.907881 – ident: ref44 doi: 10.1109/18.737524 – year: 1978 ident: ref30 publication-title: Introductory Functional Analysis With Applications – ident: ref40 doi: 10.1155/2008/784292 – year: 2016 ident: ref20 publication-title: Linear and Nonlinear Programming doi: 10.1007/978-3-319-18842-3 – volume: 23 start-page: 226 year: 2010 ident: ref39 article-title: Optimal learning rates for kernel conjugate gradient regression publication-title: Proc Adv Neural Inf Process Syst – year: 1985 ident: ref18 publication-title: Adaptive Signal Processing – ident: ref1 doi: 10.1090/S0002-9947-1950-0051437-7 – year: 2013 ident: ref32 publication-title: A Cause in Real Analysis – ident: ref31 doi: 10.1016/0022-247X(71)90184-3 – ident: ref8 doi: 10.1109/TSP.2004.830991 – ident: ref33 doi: 10.6028/jres.049.044 – ident: ref10 doi: 10.1109/IJCNN.2005.1556088 – ident: ref22 doi: 10.1137/1.9780898719697 – year: 2014 ident: ref19 publication-title: Adaptive Filter Theory – ident: ref27 doi: 10.1109/ICASSP.2017.7952666 – ident: ref13 doi: 10.1109/TSP.2009.2022007 – ident: ref4 doi: 10.1162/089976698300017467 – ident: ref42 doi: 10.1109/TNNLS.2012.2200500 – ident: ref7 doi: 10.1109/TSP.2009.2022857 – ident: ref14 doi: 10.1109/TSP.2008.2009895 – ident: ref5 doi: 10.1109/TNNLS.2013.2285783 – ident: ref17 doi: 10.1109/TSP.2015.2437835 – volume: 3 start-page: 1 year: 2002 ident: ref6 article-title: Kernel independent component analysis publication-title: J Mach Learn Res – ident: ref25 doi: 10.1002/cem.728 – volume: 2 start-page: 97 year: 2001 ident: ref26 article-title: Kernel partial least squares regression in reproducing kernel Hilbert spaces publication-title: J Mach Learn Res – ident: ref15 doi: 10.1109/TSP.2010.2096420 – ident: ref9 doi: 10.1109/TSP.2004.830985 – year: 2003 ident: ref37 article-title: Robust implementations of the multistage Wiener filter – ident: ref34 doi: 10.1145/355984.355989 – ident: ref3 doi: 10.1007/978-1-4757-3264-1 – year: 2013 ident: ref35 publication-title: Matrix Computations doi: 10.56021/9781421407944 – year: 2002 ident: ref2 publication-title: Learning With Kernels Support Vector Machines Regularization Optimization and Beyond – ident: ref16 doi: 10.1109/TNNLS.2011.2178446 – ident: ref28 doi: 10.1109/TSP.2015.2457396 – ident: ref41 doi: 10.1109/TNNLS.2013.2258936 – year: 1987 ident: ref23 publication-title: Practical Methods of Optimization – ident: ref38 doi: 10.1109/TSP.2015.2493990 – start-page: 1017 year: 2007 ident: ref24 article-title: Kernel conjugate gradient for fast kernel machines publication-title: Proc Int Joint Conf Artif Intell – ident: ref36 doi: 10.1109/TSP.2016.2605075 – ident: ref29 doi: 10.1109/TSP.2017.2726991
SSID	ssj0014496
Score	2.4486089
Snippet	Kernel methods have been successfully applied to nonlinear problems in machine learning and signal processing. Various kernel-based algorithms have been...
SourceID	proquest crossref ieee
SourceType	Aggregation Database Enrichment Source Index Database Publisher
StartPage	4377
SubjectTerms	Adaptive algorithms Algorithms Computational efficiency conjugate gradient algorithm Conjugates Convergence Criteria Hilbert space Iterative methods Kernel Machine learning Machine learning algorithms nonlinear processing online sparsification Optimization regularization Reproducing kernel Hilbert space Robustness (mathematics) Signal processing Signal processing algorithms
Title	The Kernel Conjugate Gradient Algorithms
URI	https://ieeexplore.ieee.org/document/8408777 https://www.proquest.com/docview/2117059955
Volume	66
WOSCitedRecordID	wos000439360200001&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D
hasFullText	1
inHoldings	1
isFullTextHit
isPrint
journalDatabaseRights	– providerCode: PRVIEE databaseName: IEEE/IET Electronic Library (IEL) (UW System Shared) customDbUrl: eissn: 1941-0476 dateEnd: 99991231 omitProxy: false ssIdentifier: ssj0014496 issn: 1053-587X databaseCode: RIE dateStart: 19910101 isFulltext: true titleUrlDefault: https://ieeexplore.ieee.org/ providerName: IEEE
link	http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV1bS8MwFD7M4YM-eJvidEoffFCwrm2SpXkcwwsoY6DC3kqbnOhkttJ1_n6TrBuCIvjWQk4pXy7nmvMBnBGmMsm1cUsYKp-aNz-NUuoTqu3W4kbMsZY88OEwHo_FqAGXq7swiOiKz_DKPrpcvirk3IbKusYZse3r1mCN897irtYqY0Cp4-Iy5gLxWczHy5RkILpPjyNbwxVfRUY3udLDbyrIcar8OIiddrnZ_t9_7cBWbUV6_cW070ID8z3Y_NZbsAXnZgF491jmOPUGRf42t_Ey77Z0JV6V15--FOWken2f7cPzzfXT4M6vaRF8GYmw8jPKpKUcigKiQx0LzjIhaZBiiAFLtRaIaUCM5jfGC1eBkMhUTMNYkSzVkiM5gGZe5HgIHmNIFBFaGCePplJkkVBa9USmQ5kyotrQXSKVyLpnuKWumCbOdwhEYrBNLLZJjW0bLlYSH4t-GX-MbVksV-NqGNvQWU5GUm-oWRJZhhzbHI0d_S51DBv22zbcG7IONKtyjiewLj-ryaw8dWvlCz6Gu6I
linkProvider	IEEE
linkToHtml	http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV1bS8MwFD54A_XBuzivffBBwW5tk9jmUcQbm0Nwwt5Km5zoZLbSdf5-k6wrA0XwrYUcWr5czjXnAzglTKYiVNotYShdqt_cJEioS6gyWyvUYpa1pBN2u1G_z5_m4KK-C4OItvgMm-bR5vJlLsYmVNbSzohpXzcPi4zSwJvc1qpzBpRaNi5tMBCXRWF_mpT0eKv3_GSquKJmoLWTLT6cUUKWVeXHUWz1y-36__5sA9YqO9K5mkz8JsxhtgWrM90Ft-FMLwGnjUWGQ-c6z97HJmLm3BW2yKt0roaveTEo3z5GO_Bye9O7vncrYgRXBNwv3ZQyYUiHAo8oX0U8ZCkX1EvQR48lSnHExCNa92vzJZQeF8hkRP1IkjRRIkSyCwtZnuEeOIwhkYQrrt08mgieBlwqeclT5YuEEdmA1hSpWFRdww15xTC23oPHY41tbLCNK2wbcF5LfE46ZvwxdttgWY-rYGzA4XQy4mpLjeLAcOSY9mhs_3epE1i-7z124s5Dt30AK-Y7Jvjrs0NYKIsxHsGS-CoHo-LYrptvLyC-6Q
openUrl	ctx_ver=Z39.88-2004&ctx_enc=info%3Aofi%2Fenc%3AUTF-8&rfr_id=info%3Asid%2Fsummon.serialssolutions.com&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=The+Kernel+Conjugate+Gradient+Algorithms&rft.jtitle=IEEE+transactions+on+signal+processing&rft.au=Zhang%2C+Ming&rft.au=Wang%2C+Xiaojian&rft.au=Chen%2C+Xiaoming&rft.au=Zhang%2C+Anxue&rft.date=2018-08-15&rft.pub=IEEE&rft.issn=1053-587X&rft.volume=66&rft.issue=16&rft.spage=4377&rft.epage=4387&rft_id=info:doi/10.1109%2FTSP.2018.2853109&rft.externalDocID=8408777
thumbnail_l	http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/lc.gif&issn=1053-587X&client=summon
thumbnail_m	http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/mc.gif&issn=1053-587X&client=summon
thumbnail_s	http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/sc.gif&issn=1053-587X&client=summon