Comparison of \ell -Norm SVR and Sparse Coding Algorithms for Linear Regression

Support vector regression (SVR) is a popular function estimation technique based on Vapnik's concept of support vector machine. Among many variants, the l 1 -norm SVR is known to be good at selecting useful features when the features are redundant. Sparse coding (SC) is a technique widely used...

Full description

Saved in:
Bibliographic Details
Published in:IEEE transaction on neural networks and learning systems Vol. 26; no. 8; pp. 1828 - 1833
Main Authors: Qingtian Zhang, Xiaolin Hu, Bo Zhang
Format: Journal Article
Language:English
Published: IEEE 01.08.2015
Subjects:
Algorithm design and analysis
Frequency selective surfaces
Matching pursuit algorithms
Newton linear programming (NLP)
radial basis function (RBF) neural network
regression
sparse coding (SC)
support vector machine (SVM)
Support vector machines
Testing
Training
Vectors
Newton linear programming (NLP)
sparse coding (SC)
regression
support vector machine (SVM)
radial basis function (RBF) neural network
ISSN:2162-237X, 2162-2388
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Abstract Support vector regression (SVR) is a popular function estimation technique based on Vapnik's concept of support vector machine. Among many variants, the l 1 -norm SVR is known to be good at selecting useful features when the features are redundant. Sparse coding (SC) is a technique widely used in many areas and a number of efficient algorithms are available. Both l 1 -norm SVR and SC can be used for linear regression. In this brief, the close connection between the l 1 -norm SVR and SC is revealed and some typical algorithms are compared for linear regression. The results show that the SC algorithms outperform the Newton linear programming algorithm, an efficient l 1 -norm SVR algorithm, in efficiency. The algorithms are then used to design the radial basis function (RBF) neural networks. Experiments on some benchmark data sets demonstrate the high efficiency of the SC algorithms. In particular, one of the SC algorithms, the orthogonal matching pursuit is two orders of magnitude faster than a well-known RBF network designing algorithm, the orthogonal least squares algorithm.
AbstractList Support vector regression (SVR) is a popular function estimation technique based on Vapnik's concept of support vector machine. Among many variants, the l 1 -norm SVR is known to be good at selecting useful features when the features are redundant. Sparse coding (SC) is a technique widely used in many areas and a number of efficient algorithms are available. Both l 1 -norm SVR and SC can be used for linear regression. In this brief, the close connection between the l 1 -norm SVR and SC is revealed and some typical algorithms are compared for linear regression. The results show that the SC algorithms outperform the Newton linear programming algorithm, an efficient l 1 -norm SVR algorithm, in efficiency. The algorithms are then used to design the radial basis function (RBF) neural networks. Experiments on some benchmark data sets demonstrate the high efficiency of the SC algorithms. In particular, one of the SC algorithms, the orthogonal matching pursuit is two orders of magnitude faster than a well-known RBF network designing algorithm, the orthogonal least squares algorithm.
Author Xiaolin Hu
Qingtian Zhang
Bo Zhang
Author_xml – sequence: 1
  surname: Qingtian Zhang
  fullname: Qingtian Zhang
  email: forgettingzqt@yahoo.cn
  organization: Dept. of Comput. Sci. & Technol., Tsinghua Univ., Beijing, China
– sequence: 2
  surname: Xiaolin Hu
  fullname: Xiaolin Hu
  email: xlhu@tsinghua.edu.cn
  organization: Dept. of Comput. Sci. & Technol., Tsinghua Univ., Beijing, China
– sequence: 3
  surname: Bo Zhang
  fullname: Bo Zhang
  email: dcszb@tsinghua.edu.cn
  organization: Dept. of Comput. Sci. & Technol., Tsinghua Univ., Beijing, China
BookMark eNp9kM9qwzAMh83oWLuuL7Bd_ALpLCd2kmMJ-wehhbYbOwyCnSidRxIXO5e9_dK19LDDdJFAfNKP75qMOtshIbfA5gAsvd8ul_lmzhlEcx7GMY_EBZlwkDzgYZKMznP8PiYz77_YUJIJGaVXZMyFCDmkYkJWmW33yhlvO2pr-oFNQ4OldS3dvK2p6iq6GdYeaWYr0-3ootlZZ_rP1tPaOpqbDpWja9w59N7Y7oZc1qrxODv1KXl9fNhmz0G-enrJFnlQAnAYcikNoEBwplRVM2QVclZVGJcgtIJYRVwixEJqQAi10FCGutISNSZM8XBK-PFu6az3Duti70yr3HcBrDgIKn4FFQdBxUnQACV_oNL0qh9i906Z5n_07ogaRDz_kmkiJIfwB1ltc-k
CODEN ITNNAL
CitedBy_id crossref_primary_10_1109_TNNLS_2016_2573644
crossref_primary_10_1109_ACCESS_2019_2948059
crossref_primary_10_7554_eLife_56212
crossref_primary_10_1109_TNNLS_2017_2731319
crossref_primary_10_1111_exsy_13471
crossref_primary_10_1177_09544089221124288
crossref_primary_10_1109_ACCESS_2019_2945807
crossref_primary_10_1088_0031_9155_61_19_7162
crossref_primary_10_1016_j_jfranklin_2025_107798
crossref_primary_10_1186_s12938_017_0342_y
crossref_primary_10_1109_ACCESS_2021_3054823
crossref_primary_10_1016_j_neunet_2024_106633
crossref_primary_10_1016_j_neucom_2018_09_089
crossref_primary_10_1109_TNNLS_2022_3192065
Cites_doi 10.1023/B:STCO.0000035301.49549.88
10.1017/CBO9780511543241
10.1109/72.80341
10.1137/S1064827596304010
10.1016/j.neucom.2007.12.009
10.1214/009053604000000067
10.1002/cpa.20132
10.1007/978-1-4613-9617-8_2
10.1162/neco.1991.3.2.246
10.1093/comjnl/7.4.308
10.1038/381607a0
10.1093/imanum/20.3.389
10.1137/090777761
10.1109/ICASSP.2005.1416408
10.1109/78.258082
10.1109/TNN.2006.880860
10.1109/ACSSC.1993.342465
10.1023/B:COAP.0000026884.66338.df
10.1080/00207720802083018
ContentType Journal Article
DBID 97E
RIA
RIE
AAYXX
CITATION
DOI 10.1109/TNNLS.2014.2377245
DatabaseName IEEE Xplore (IEEE)
IEEE All-Society Periodicals Package (ASPP) 1998–Present
IEEE Electronic Library (IEL)
CrossRef
DatabaseTitle CrossRef
DatabaseTitleList
Database_xml – sequence: 1
  dbid: RIE
  name: IEEE Electronic Library (IEL)
  url: https://ieeexplore.ieee.org/
  sourceTypes: Publisher
DeliveryMethod fulltext_linktorsrc
Discipline Computer Science
EISSN 2162-2388
EndPage 1833
ExternalDocumentID 10_1109_TNNLS_2014_2377245
6985621
Genre orig-research
GrantInformation_xml – fundername: Natural Science Foundation of Beijing
  grantid: 4132046
– fundername: National Natural Science Foundation of China
  grantid: 61273023
  funderid: 10.13039/501100001809
– fundername: National Basic Research Program (973 Program) of China
  grantid: 2013CB329403; 2012CB316301
– fundername: Tsinghua National Laboratory for Information Science and Technology Cross-Discipline Foundation
GroupedDBID 0R~
4.4
5VS
6IK
97E
AAJGR
AARMG
AASAJ
AAWTH
ABAZT
ABQJQ
ABVLG
ACIWK
ACPRK
AENEX
AFRAH
AGQYO
AGSQL
AHBIQ
AKJIK
AKQYR
ALMA_UNASSIGNED_HOLDINGS
ATWAV
BEFXN
BFFAM
BGNUA
BKEBE
BPEOZ
EBS
EJD
IFIPE
IPLJI
JAVBF
M43
MS~
O9-
OCL
PQQKQ
RIA
RIE
RNS
AAYXX
CITATION
ID FETCH-LOGICAL-c1121-23ab11a1520aadf0e0de20dde7c15ba17a426e1756b1e13b5b1c3bdb6ebe80a23
IEDL.DBID RIE
ISSN 2162-237X
IngestDate Tue Nov 18 21:49:20 EST 2025
Sat Nov 29 01:39:51 EST 2025
Tue Aug 26 16:37:37 EDT 2025
IsPeerReviewed false
IsScholarly true
Issue 8
Keywords Newton linear programming (NLP)
sparse coding (SC)
regression
support vector machine (SVM)
radial basis function (RBF) neural network
Language English
License https://ieeexplore.ieee.org/Xplorehelp/downloads/license-information/IEEE.html
LinkModel DirectLink
MergedId FETCHMERGED-LOGICAL-c1121-23ab11a1520aadf0e0de20dde7c15ba17a426e1756b1e13b5b1c3bdb6ebe80a23
PMID 25532195
PageCount 6
ParticipantIDs ieee_primary_6985621
crossref_citationtrail_10_1109_TNNLS_2014_2377245
crossref_primary_10_1109_TNNLS_2014_2377245
PublicationCentury 2000
PublicationDate 2015-Aug.
2015-8-00
PublicationDateYYYYMMDD 2015-08-01
PublicationDate_xml – month: 08
  year: 2015
  text: 2015-Aug.
PublicationDecade 2010
PublicationTitle IEEE transaction on neural networks and learning systems
PublicationTitleAbbrev TNNLS
PublicationYear 2015
Publisher IEEE
Publisher_xml – name: IEEE
References ref13
ref12
ref15
ref14
mangasarian (ref9) 2006; 7
ref31
ref30
ref11
ref10
ref19
efron (ref25) 2004; 32
lee (ref16) 2007; 19
müller (ref5) 1997
bache (ref32) 2013
stitson (ref6) 1999
ref24
ref23
vapnik (ref1) 1963; 24
ref22
ref21
yang (ref17) 2010
schölkopf (ref4) 1998
ref28
ref27
ref29
perkins (ref26) 2003; 20
ref8
ref7
schölkopf (ref3) 1996
ng (ref20) 2004
yang (ref18) 2009
vapnik (ref2) 1964; 25
References_xml – volume: 7
  start-page: 1517
  year: 2006
  ident: ref9
  article-title: Exact 1-norm support vector machines via unconstrained convex differentiable minimization
  publication-title: J Mach Learn Res
– start-page: 999
  year: 1997
  ident: ref5
  article-title: Predicting time series with support vector machines
  publication-title: Proc Int Conf Artif Neural Netw
– volume: 24
  start-page: 774
  year: 1963
  ident: ref1
  article-title: Pattern recognition using generalized portrait method
  publication-title: Autom Remote Control
– ident: ref19
  doi: 10.1023/B:STCO.0000035301.49549.88
– ident: ref28
  doi: 10.1017/CBO9780511543241
– ident: ref29
  doi: 10.1109/72.80341
– ident: ref12
  doi: 10.1137/S1064827596304010
– volume: 20
  start-page: 592
  year: 2003
  ident: ref26
  article-title: Online feature selection using grafting
  publication-title: Proc 20th Int Conf Mach Learn
– ident: ref22
  doi: 10.1016/j.neucom.2007.12.009
– year: 2013
  ident: ref32
  publication-title: UCI Machine Learning Repository
– volume: 32
  start-page: 407
  year: 2004
  ident: ref25
  article-title: Least angle regression
  publication-title: Ann Statist
  doi: 10.1214/009053604000000067
– start-page: 1849
  year: 2010
  ident: ref17
  article-title: Fast $\ell 1$ -minimization algorithms and an application in robust face recognition: A review
  publication-title: Proc Int Conf Image Process
– start-page: 47
  year: 1996
  ident: ref3
  article-title: Incorporating invariances in support vector learning machines
  publication-title: Proc Int Conf Artif Neural Netw
– ident: ref15
  doi: 10.1002/cpa.20132
– ident: ref8
  doi: 10.1007/978-1-4613-9617-8_2
– ident: ref27
  doi: 10.1162/neco.1991.3.2.246
– ident: ref7
  doi: 10.1093/comjnl/7.4.308
– start-page: 78
  year: 2004
  ident: ref20
  article-title: Feature selection, $L_{1}$ vs. $L_{2}$ regularization, and rotational invariance
  publication-title: Proc 21st Int Conf Mach Learn
– ident: ref10
  doi: 10.1038/381607a0
– ident: ref13
  doi: 10.1093/imanum/20.3.389
– volume: 25
  start-page: 821
  year: 1964
  ident: ref2
  article-title: A note on one class of perceptrons
  publication-title: Autom Remote Control
– ident: ref24
  doi: 10.1137/090777761
– ident: ref14
  doi: 10.1109/ICASSP.2005.1416408
– start-page: 1794
  year: 2009
  ident: ref18
  article-title: Linear spatial pyramid matching using sparse coding for image classification
  publication-title: Proc IEEE Conf Comput Vis Pattern Recognit
– start-page: 285
  year: 1999
  ident: ref6
  article-title: Support vector regression with ANOVA decomposition kernels
  publication-title: Advances in Kernel Methods Support Vector Learning
– ident: ref23
  doi: 10.1109/78.258082
– ident: ref31
  doi: 10.1109/TNN.2006.880860
– ident: ref11
  doi: 10.1109/ACSSC.1993.342465
– volume: 19
  start-page: 801
  year: 2007
  ident: ref16
  article-title: Efficient sparse coding algorithms
  publication-title: Proc Adv Neural Inf Process Syst
– ident: ref21
  doi: 10.1023/B:COAP.0000026884.66338.df
– ident: ref30
  doi: 10.1080/00207720802083018
– start-page: 640
  year: 1998
  ident: ref4
  article-title: Prior knowledge in support vector kernels
  publication-title: Proc Adv Neural Inf Process Syst
SSID ssj0000605649
Score 2.1398182
Snippet Support vector regression (SVR) is a popular function estimation technique based on Vapnik's concept of support vector machine. Among many variants, the l 1...
SourceID crossref
ieee
SourceType Enrichment Source
Index Database
Publisher
StartPage 1828
SubjectTerms Algorithm design and analysis
Frequency selective surfaces
Matching pursuit algorithms
Newton linear programming (NLP)
radial basis function (RBF) neural network
regression
sparse coding (SC)
support vector machine (SVM)
Support vector machines
Testing
Training
Vectors
Title Comparison of \ell -Norm SVR and Sparse Coding Algorithms for Linear Regression
URI https://ieeexplore.ieee.org/document/6985621
Volume 26
hasFullText 1
inHoldings 1
isFullTextHit
isPrint
journalDatabaseRights – providerCode: PRVIEE
  databaseName: IEEE Electronic Library (IEL)
  customDbUrl:
  eissn: 2162-2388
  dateEnd: 99991231
  omitProxy: false
  ssIdentifier: ssj0000605649
  issn: 2162-237X
  databaseCode: RIE
  dateStart: 20120101
  isFulltext: true
  titleUrlDefault: https://ieeexplore.ieee.org/
  providerName: IEEE
link http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV1LS8NAEF5UPHjxVcX6Yg_eNO0-kmxylGLxIFGslh6EsLuZ1EJMpK3-fiePBgURvIXsLoT5Mvm-2czOEHJhtcdBKuGkMk0clyMWOgwTB8Mh17gBM6IqPD--U1EUTCbhwxq5as_CAECVfAa98rL6l58U9qPcKuv7YYB0jbHOulJ-fVar3U9hqMv9Su0K7gtHSDVZnZFhYf8piu5GZSKX28MRJdyyYw2qaYkO6_2gpG89ViqKGe787-F2yXYjJel1jf0eWYN8n-ys2jTQxms75H7Q9hqkRUpfIMuoE6FWpaPxI9V5Qkc4vAA6KEoio9fZtJjPlq9vC4qKlmK0it5AH2Fap8zmB-R5ePM0uHWaPgqORTXF0RbacK6RqZnWScqAJSAYfteU5Z7RXGmkaUAd4RsOXBrPcCtNYnwEOGBayEOykRc5HBEquTFIeVoGFrWgEtpCYn1jmbAIroQu4StTxrYpMl72usjiKthgYVwhEZdIxA0SXXLZrnmvS2z8ObtT2r6d2Zj9-PfbJ2QLF3t1xt4p2VjOP-CMbNrP5WwxP6_eoC9XccBz
linkProvider IEEE
linkToHtml http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV1bS8MwFA5DBX3xNsV5zYNvWs2l18cxFMVZxc2xB6Ek6dkczFa26e_3pOuKggi-lSYt5XxNv--kJ_kIOTXK4yAD4QzkIHVcjlioKEodTIdc7YZMi2Lj-V47iOOw348ea-S8WgsDAEXxGVzYw-JffpqbDztVdulHIdI15jrL1jmrXK1VzagwVOZ-oXcF94UjZNBfrJJh0WU3jtsdW8rlXmBLIFzrWYN6WuKQ9X6Q0jeXlYJkrjf-93ibZL0Uk7Q5R3-L1CDbJhsLowZajts6eWhVboM0H9AXGI-pE6NapZ3eE1VZSjvYPAXayi2V0eZ4mE9Gs9e3KUVNSzFfxfFAn2A4L5rNdsjz9VW3deOUTgqOQT3FMRZKc66Qq5lS6YABS0Ew_LIFhnta8UAhUQMqCV9z4FJ7mhupU-0jxCFTQu6SpSzPYI9QybVG0lMyNKgGA6EMpMbXhgmD8EpoEL4IZWLKbcat28U4KdINFiUFEolFIimRaJCz6pr3-SYbf_au29hXPcuw7_9--oSs3nTv20n7Nr47IGt4I29ev3dIlmaTDzgiK-ZzNppOjou36QuNZMO8
openUrl ctx_ver=Z39.88-2004&ctx_enc=info%3Aofi%2Fenc%3AUTF-8&rfr_id=info%3Asid%2Fsummon.serialssolutions.com&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Comparison+of+%24%5Cell+_%7B1%7D%24+-Norm+SVR+and+Sparse+Coding+Algorithms+for+Linear+Regression&rft.jtitle=IEEE+transaction+on+neural+networks+and+learning+systems&rft.au=Qingtian+Zhang&rft.au=Xiaolin+Hu&rft.au=Bo+Zhang&rft.date=2015-08-01&rft.issn=2162-237X&rft.eissn=2162-2388&rft.volume=26&rft.issue=8&rft.spage=1828&rft.epage=1833&rft_id=info:doi/10.1109%2FTNNLS.2014.2377245&rft.externalDBID=n%2Fa&rft.externalDocID=10_1109_TNNLS_2014_2377245
thumbnail_l http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/lc.gif&issn=2162-237X&client=summon
thumbnail_m http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/mc.gif&issn=2162-237X&client=summon
thumbnail_s http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/sc.gif&issn=2162-237X&client=summon