Easily Parallelizable and Distributable Class of Algorithms for Structured Sparsity, with Optimal Acceleration

Many statistical learning problems can be posed as minimization of a sum of two convex functions, one typically a composition of nonsmooth and linear functions. Examples include regression under structured sparsity assumptions. Popular algorithms for solving such problems, for example, ADMM, often i...

Full description

Saved in:
Bibliographic Details
Published in:Journal of computational and graphical statistics Vol. 28; no. 4; pp. 821 - 833
Main Authors: Ko, Seyoon, Yu, Donghyeon, Won, Joong-Ho
Format: Journal Article
Language:English
Published: Alexandria Taylor & Francis 02.10.2019
American Statistical Association, the Institute of Mathematical Statistics, and the Interface Foundation of North America
Taylor & Francis Ltd
Subjects:
Acceleration
Advances in Sampling and Optimization
Algorithms
Computer networks
Convergence
Distributed computing
Distributed processing
GPU
Linear functions
Machine learning
Monotone operator theory
Nonsmooth optimization
Operator splitting
Optimization
Parallel processing
Regression analysis
Sparsity
Statistical analysis
ISSN:1061-8600, 1537-2715
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Abstract Many statistical learning problems can be posed as minimization of a sum of two convex functions, one typically a composition of nonsmooth and linear functions. Examples include regression under structured sparsity assumptions. Popular algorithms for solving such problems, for example, ADMM, often involve nontrivial optimization subproblems or smoothing approximation. We consider two classes of primal-dual algorithms that do not incur these difficulties, and unify them from a perspective of monotone operator theory. From this unification, we propose a continuum of preconditioned forward-backward operator splitting algorithms amenable to parallel and distributed computing. For the entire region of convergence of the whole continuum of algorithms, we establish its rates of convergence. For some known instances of this continuum, our analysis closes the gap in theory. We further exploit the unification to propose a continuum of accelerated algorithms. We show that the whole continuum attains the theoretically optimal rate of convergence. The scalability of the proposed algorithms, as well as their convergence behavior, is demonstrated up to 1.2 million variables with a distributed implementation. The code is available at https://github.com/kose-y/dist-primal-dual . Supplemental materials for this article are available online.
AbstractList Many statistical learning problems can be posed as minimization of a sum of two convex functions, one typically a composition of nonsmooth and linear functions. Examples include regression under structured sparsity assumptions. Popular algorithms for solving such problems, for example, ADMM, often involve nontrivial optimization subproblems or smoothing approximation. We consider two classes of primal-dual algorithms that do not incur these difficulties, and unify them from a perspective of monotone operator theory. From this unification, we propose a continuum of preconditioned forward-backward operator splitting algorithms amenable to parallel and distributed computing. For the entire region of convergence of the whole continuum of algorithms, we establish its rates of convergence. For some known instances of this continuum, our analysis closes the gap in theory. We further exploit the unification to propose a continuum of accelerated algorithms. We show that the whole continuum attains the theoretically optimal rate of convergence. The scalability of the proposed algorithms, as well as their convergence behavior, is demonstrated up to 1.2 million variables with a distributed implementation. The code is available at https://github.com/kose-y/dist-primal-dual . Supplemental materials for this article are available online.
Many statistical learning problems can be posed as minimization of a sum of two convex functions, one typically a composition of nonsmooth and linear functions. Examples include regression under structured sparsity assumptions. Popular algorithms for solving such problems, for example, ADMM, often involve nontrivial optimization subproblems or smoothing approximation. We consider two classes of primal-dual algorithms that do not incur these difficulties, and unify them from a perspective of monotone operator theory. From this unification, we propose a continuum of preconditioned forward-backward operator splitting algorithms amenable to parallel and distributed computing. For the entire region of convergence of the whole continuum of algorithms, we establish its rates of convergence. For some known instances of this continuum, our analysis closes the gap in theory. We further exploit the unification to propose a continuum of accelerated algorithms. We show that the whole continuum attains the theoretically optimal rate of convergence. The scalability of the proposed algorithms, as well as their convergence behavior, is demonstrated up to 1.2 million variables with a distributed implementation. The code is available at https://github.com/kose-y/dist-primal-dual. Supplemental materials for this article are available online.
Author Won, Joong-Ho
Yu, Donghyeon
Ko, Seyoon
Author_xml – sequence: 1
  givenname: Seyoon
  surname: Ko
  fullname: Ko, Seyoon
  organization: Department of Statistics, Seoul National University
– sequence: 2
  givenname: Donghyeon
  surname: Yu
  fullname: Yu, Donghyeon
  organization: Department of Statistics, Inha University
– sequence: 3
  givenname: Joong-Ho
  surname: Won
  fullname: Won, Joong-Ho
  email: wonj@stats.snu.ac.kr
  organization: Department of Statistics, Seoul National University
BookMark eNqFkFFrFDEQx4NUsK1-BCEgvrlnJrnd7OKLx9mqUGih9TnMZrOaI7c5J1nK-emb61UQH_QpIfP7ZWb-Z-xkipNj7DWIBYhWvAfRQNsIsZACugXUndS1fsZOoVa6khrqk3IvTHWAXrCzlDZCCGg6fcqmC0w-7PkNEobggv-FfXAcp4F_8imT7-f8-LIOmBKPI1-F75F8_rFNfIzEbzPNNs_kBn67Q0o-79_x-1Ln17vstxj4yloXHGH2cXrJno8Yknv1dJ6zb5cXd-sv1dX156_r1VVlZatyBTgg6M4KsP1SWwTXgW5dGXoYQbhla4dGDq3qlJCql1aNTi9tN_aqdbWtnTpnb47_7ij-nF3KZhNnmkpLI5USxWq0KtSHI2UppkRuNNbnxzkzoQ8GhDkEbH4HbA4Bm6eAi13_Ze-oLEz7_3pvj94m5Uh_SlIJbZa1bKWUULiPR85PJegt3kcKg8m4D5FGwsn6ZNS_Wz0ALn2g1w
CitedBy_id crossref_primary_10_1007_s10444_020_09840_9
crossref_primary_10_1093_bib_bbab256
crossref_primary_10_1137_18M1207685
crossref_primary_10_1214_21_STS835
Cites_doi 10.1088/0266-5611/29/2/025011
10.1214/07-AOAS131
10.1561/2200000016
10.1080/10618600.2015.1114491
10.1137/050626090
10.1080/10618600.2013.878662
10.1137/100801652
10.1109/TMI.2010.2093536
10.1145/1553374.1553431
10.1214/11-AOS878
10.1016/0167-2789(92)90242-F
10.1016/j.orl.2015.02.001
10.1137/151003076
10.1137/09076934X
10.1214/11-AOAS514
10.1186/s13663-016-0543-2
10.1111/j.1467-9868.2005.00532.x
10.1007/s10957-012-0245-9
10.1137/080725891
10.1137/100814494
10.1137/080716542
10.1007/s10107-004-0552-5
10.1111/j.1467-9868.2005.00490.x
10.1007/s10444-011-9254-8
10.1007/s10851-010-0251-1
10.1007/s10107-015-0957-3
10.1007/s11075-015-0007-5
10.1145/1835804.1835847
10.1088/0266-5611/27/12/125007
10.1007/978-1-4419-9467-7
10.1080/02331934.2014.966306
10.1137/130919362
10.1007/978-1-4419-8853-9
10.1088/0031-9155/57/10/3065
10.1007/s11228-011-0191-y
10.1109/ICIP.2014.7025841
10.1080/10618600.2017.1328363
ContentType Journal Article
Copyright 2019 American Statistical Association, Institute of Mathematical Statistics, and Interface Foundation of North America 2019
2019 American Statistical Association, the Institute of Mathematical Statistics, and the Interface Foundation of North America
2019 American Statistical Association, Institute of Mathematical Statistics, and Interface Foundation of North America
Copyright_xml – notice: 2019 American Statistical Association, Institute of Mathematical Statistics, and Interface Foundation of North America 2019
– notice: 2019 American Statistical Association, the Institute of Mathematical Statistics, and the Interface Foundation of North America
– notice: 2019 American Statistical Association, Institute of Mathematical Statistics, and Interface Foundation of North America
DBID AAYXX
CITATION
JQ2
DOI 10.1080/10618600.2019.1592757
DatabaseName CrossRef
ProQuest Computer Science Collection
DatabaseTitle CrossRef
ProQuest Computer Science Collection
DatabaseTitleList

ProQuest Computer Science Collection
DeliveryMethod fulltext_linktorsrc
Discipline Statistics
Mathematics
EISSN 1537-2715
EndPage 833
ExternalDocumentID 10_1080_10618600_2019_1592757
45282221
1592757
Genre Article
GrantInformation_xml – fundername: National Research Foundation of Korea
GroupedDBID -~X
.4S
.7F
.DC
.QJ
0BK
0R~
2AX
30N
4.4
5GY
AAENE
AAGDL
AAHIA
AAJMT
AALDU
AAMIU
AAPUL
AAQRR
ABBHK
ABCCY
ABFAN
ABFIM
ABJNI
ABLIJ
ABLJU
ABPAQ
ABPEM
ABQDR
ABTAI
ABXUL
ABXYU
ABYWD
ACGFO
ACGFS
ACIWK
ACMTB
ACTIO
ACTMH
ADCVX
ADGTB
ADODI
ADXHL
AEGXH
AELLO
AENEX
AEOZL
AEPSL
AEUPB
AEYOC
AFRVT
AFVYC
AGDLA
AGMYJ
AHDZW
AIAGR
AIJEM
AKBRZ
AKBVH
AKOOK
ALMA_UNASSIGNED_HOLDINGS
ALQZU
ALRMG
AMVHM
AQRUH
AQTUD
ARCSS
AVBZW
AWYRJ
BLEHA
CCCUG
CS3
D0L
DGEBU
DKSSO
DQDLB
DSRWC
DU5
EBS
ECEWR
E~A
E~B
F5P
GTTXZ
H13
HF~
HQ6
HZ~
H~P
IPNFZ
IPSME
J.P
JAA
JAAYA
JBMMH
JBZCM
JENOY
JHFFW
JKQEH
JLEZI
JLXEF
JMS
JPL
JST
KYCEM
LJTGL
M4Z
MS~
NA5
NY~
O9-
P2P
PQQKQ
RIG
RNANH
ROSJB
RTWRZ
RWL
RXW
S-T
SA0
SNACF
TAE
TASJS
TBQAZ
TDBHL
TEJ
TFL
TFT
TFW
TN5
TTHFI
TUROJ
TUS
UT5
UU3
WZA
XWC
ZGOLN
~S~
ADYSH
AMPGV
AAYXX
CITATION
JQ2
ID FETCH-LOGICAL-c283t-1ada179c01cb47ca1e9178e001df10e48cd62d8393023b2c3fe74c9fb38e5c5e3
IEDL.DBID TFW
ISSN 1061-8600
IngestDate Wed Aug 13 07:40:43 EDT 2025
Tue Nov 18 22:27:32 EST 2025
Sat Nov 29 03:24:17 EST 2025
Thu May 29 08:47:49 EDT 2025
Mon Oct 20 23:49:06 EDT 2025
IsPeerReviewed true
IsScholarly true
Issue 4
Language English
LinkModel DirectLink
MergedId FETCHMERGED-LOGICAL-c283t-1ada179c01cb47ca1e9178e001df10e48cd62d8393023b2c3fe74c9fb38e5c5e3
Notes ObjectType-Article-1
SourceType-Scholarly Journals-1
ObjectType-Feature-2
content type line 14
PQID 2330023673
PQPubID 29738
PageCount 13
ParticipantIDs proquest_journals_2330023673
jstor_primary_10_2307_45282221
crossref_primary_10_1080_10618600_2019_1592757
crossref_citationtrail_10_1080_10618600_2019_1592757
informaworld_taylorfrancis_310_1080_10618600_2019_1592757
PublicationCentury 2000
PublicationDate 2019-10-02
PublicationDateYYYYMMDD 2019-10-02
PublicationDate_xml – month: 10
  year: 2019
  text: 2019-10-02
  day: 02
PublicationDecade 2010
PublicationPlace Alexandria
PublicationPlace_xml – name: Alexandria
PublicationTitle Journal of computational and graphical statistics
PublicationYear 2019
Publisher Taylor & Francis
American Statistical Association, the Institute of Mathematical Statistics, and the Interface Foundation of North America
Taylor & Francis Ltd
Publisher_xml – name: Taylor & Francis
– name: American Statistical Association, the Institute of Mathematical Statistics, and the Interface Foundation of North America
– name: Taylor & Francis Ltd
References CIT0030
Combettes P. L. (CIT0016) 2014
CIT0010
CIT0032
CIT0031
CIT0012
CIT0034
CIT0011
CIT0033
CIT0014
CIT0036
CIT0013
CIT0035
Bertsekas D. P (CIT0004) 2009
CIT0015
CIT0037
CIT0018
CIT0017
CIT0039
CIT0019
CIT0041
CIT0040
CIT0021
CIT0020
CIT0042
CIT0001
CIT0023
CIT0022
Xin B. (CIT0038) 2014
Lin Z. (CIT0026) 2011
CIT0003
CIT0025
CIT0002
CIT0024
CIT0005
CIT0027
CIT0007
CIT0029
CIT0006
CIT0028
CIT0009
CIT0008
References_xml – ident: CIT0010
  doi: 10.1088/0266-5611/29/2/025011
– ident: CIT0021
  doi: 10.1214/07-AOAS131
– ident: CIT0007
  doi: 10.1561/2200000016
– ident: CIT0041
  doi: 10.1080/10618600.2015.1114491
– ident: CIT0042
– ident: CIT0015
  doi: 10.1137/050626090
– ident: CIT0039
  doi: 10.1080/10618600.2013.878662
– ident: CIT0029
  doi: 10.1137/100801652
– ident: CIT0032
  doi: 10.1109/TMI.2010.2093536
– ident: CIT0024
  doi: 10.1145/1553374.1553431
– ident: CIT0036
  doi: 10.1214/11-AOS878
– ident: CIT0033
  doi: 10.1016/0167-2789(92)90242-F
– ident: CIT0019
  doi: 10.1016/j.orl.2015.02.001
– ident: CIT0018
  doi: 10.1137/151003076
– ident: CIT0020
  doi: 10.1137/09076934X
– volume-title: Convex Optimization Theory
  year: 2009
  ident: CIT0004
– ident: CIT0012
  doi: 10.1214/11-AOAS514
– start-page: 612
  volume-title: Advances in Neural Information Processing Systems
  year: 2011
  ident: CIT0026
– ident: CIT0011
  doi: 10.1186/s13663-016-0543-2
– ident: CIT0040
  doi: 10.1111/j.1467-9868.2005.00532.x
– ident: CIT0017
  doi: 10.1007/s10957-012-0245-9
– ident: CIT0022
  doi: 10.1137/080725891
– ident: CIT0023
  doi: 10.1137/100814494
– ident: CIT0003
  doi: 10.1137/080716542
– ident: CIT0031
  doi: 10.1007/s10107-004-0552-5
– ident: CIT0035
  doi: 10.1111/j.1467-9868.2005.00490.x
– ident: CIT0037
  doi: 10.1007/s10444-011-9254-8
– ident: CIT0008
  doi: 10.1007/s10851-010-0251-1
– ident: CIT0009
  doi: 10.1007/s10107-015-0957-3
– ident: CIT0006
  doi: 10.1007/s11075-015-0007-5
– ident: CIT0027
  doi: 10.1145/1835804.1835847
– ident: CIT0028
  doi: 10.1088/0266-5611/27/12/125007
– ident: CIT0002
  doi: 10.1007/978-1-4419-9467-7
– ident: CIT0005
  doi: 10.1080/02331934.2014.966306
– ident: CIT0013
  doi: 10.1137/130919362
– ident: CIT0001
– ident: CIT0030
  doi: 10.1007/978-1-4419-8853-9
– start-page: 2163
  volume-title: AAAI
  year: 2014
  ident: CIT0038
– ident: CIT0034
  doi: 10.1088/0031-9155/57/10/3065
– ident: CIT0014
  doi: 10.1007/s11228-011-0191-y
– start-page: 4141
  volume-title: 2014 IEEE International Conference on Image Processing (ICIP)
  year: 2014
  ident: CIT0016
  doi: 10.1109/ICIP.2014.7025841
– ident: CIT0025
  doi: 10.1080/10618600.2017.1328363
SSID ssj0001697
Score 2.2975013
Snippet Many statistical learning problems can be posed as minimization of a sum of two convex functions, one typically a composition of nonsmooth and linear...
SourceID proquest
crossref
jstor
informaworld
SourceType Aggregation Database
Enrichment Source
Index Database
Publisher
StartPage 821
SubjectTerms Acceleration
Advances in Sampling and Optimization
Algorithms
Computer networks
Convergence
Distributed computing
Distributed processing
GPU
Linear functions
Machine learning
Monotone operator theory
Nonsmooth optimization
Operator splitting
Optimization
Parallel processing
Regression analysis
Sparsity
Statistical analysis
Title Easily Parallelizable and Distributable Class of Algorithms for Structured Sparsity, with Optimal Acceleration
URI https://www.tandfonline.com/doi/abs/10.1080/10618600.2019.1592757
https://www.jstor.org/stable/45282221
https://www.proquest.com/docview/2330023673
Volume 28
hasFullText 1
inHoldings 1
isFullTextHit
isPrint
journalDatabaseRights – providerCode: PRVAWR
  databaseName: Taylor & Francis Journals Complete
  customDbUrl:
  eissn: 1537-2715
  dateEnd: 99991231
  omitProxy: false
  ssIdentifier: ssj0001697
  issn: 1061-8600
  databaseCode: TFW
  dateStart: 19920301
  isFulltext: true
  titleUrlDefault: https://www.tandfonline.com
  providerName: Taylor & Francis
link http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwrV3NS8MwFA8iHubBj6k4P0YOHu1cmrRNj6IOD34MprhbSNNUB3OTdQr-976XpkMR8aDHlr6mzftM-OX3CDmCgs0azeIg0TIORJSxQFpmgoxHWnApIam7g8JXyc2NHA7TvkcTlh5WiWvooiKKcLEanVtnZY2IO8FVjIREjcCstAP5OEwiPE8OqR9d8673sIjFzLdXAYkAReozPD-95Ut2-sJdWuMVv8Vsl4h66__wCxtkzVeh9LQym02yZCdNsnq9oHAtm6SBZWjF4rxFJhe6HI3faV_PsPeKg4KNLYXh6Tky72LTLHfH9dik04Kejh-ns9H86bmk8Hd04GhqX2c2p4MX7YAgxxT3gOktxKxn_BZjIANW9rhN7nsXd2eXge_UEBgoT-YB07kGzzZdZjKRgO4trAKlhfnPC9a1Qpo8DnOoxbBFURYaXthEmLTIuLSRiSzfIcuT6cTuEipkLLJEp0waWBnmsYSQEjONnERdZOdqEVFrSBlPY47dNMaKebbTem4Vzq3yc9sinYXYS8Xj8ZtA-ln9au42UIqq24niv8i2na18HgkB90pEiNoNWYsc1EakfMwo4Qnu-PwTvveHsfdJAy8d3jA8IMugXntIVswbmMys7bzjA1lrB50
linkProvider Taylor & Francis
linkToHtml http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwrV1LT8MwDLZ4ScCBN2I8c-BIYWnTNj0iYAIxBhJDcIvSNAWksaFtIPHvsdN2GkKIA1zbumljx4_oy2eAfUzYrNE88mItI0-EKfek5cZLg1CLQEoM6u6gcDNuteTDQzJ-FoZglVRD5wVRhPPVtLhpM7qCxB1RGSMxUhMyKznEgOzHYTwJ0yHGWuLPbzfuR96Ylw1WUMQjmeoUz0-v-RKfvrCXVojFb17bhaLG4n_8xBIslIkoOy4sZxkmbHcF5q9GLK6DFZijTLQgcl6F7pkePHc-2I3uU_sVhwbrWIbjs1Mi36W-We6Ka7PJejk77jz2-s_Dp5cBw99jt46p9q1vM3b7qh0W5IDRNjC7Rrf1Qt9iDAbBwiTX4K5x1j4598pmDZ7BDGXocZ1pXNymzk0qYlS_xUJQWlRAlvO6FdJkkZ9hOkZdilLfBLmNhUnyNJA2NKEN1mGq2-vaDWBCRiKNdcKlweIwiyR6lYhroiWqE0FXDUSlImVKJnNqqNFRvCQ8reZW0dyqcm5rcDgSey2oPH4TSMb1r4ZuDyUvGp6o4BfZXWcs4yMR5l6JkIC7Pq_BdmVFqnQbA3wicJT-cbD5h7H3YPa8fdVUzYvW5RbM0S0HP_S3YQpVbXdgxryj-fR33VL5BBQjC8c
linkToPdf http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwrV1LT-MwEB6xgBAcgAVWW17rwx4J1ImTOEcEVCDYUgkQ3CzHcQCptFXTXYl_z4zjVCCEOLDXJBPH4_E8rC_fAPzGhM0azZMg1TIJRJzzQFpugjyKtYikxKDufhS-SLtdeXeX9TyasPKwSqqhy5oowvlq2tyjomwQcQdUxUgM1ATMyvYxHodpnH6DOUydEzLy687t1Blz318FRQKSaX7i-eg1b8LTG_LSBrD4zmm7SNRZ-Q9zWIVln4ayw9puvsOMHazB0p8ph2u1BouUh9Y0zuswONHVY_-Z9fSYmq84LFjfMhyeHRP1LnXNcldck002LNlh_344fpw8PFUMZ8euHE_t37Et2NVIOyTIHqNDYHaJTuuJvsUYDIG1QW7ATefk-ug08K0aAoP5ySTgutC4tU2bm1ykuPgWy0BpUf9FydtWSFMkYYHJGPUoykMTlTYVJivzSNrYxDb6AbOD4cD-BCZkIvJUZ1waLA2LRKJPSbgmUqI20XO1QDQrpIznMad2Gn3FPd1po1tFulVety3Yn4qNaiKPzwSy18uvJu4EpazbnajoE9ldZyuvRyLEvRIxwXZD3oLtxoiUdxoVPhE5Qv802vzC2L9goXfcURdn3fMtWKQ7DnsYbsMsrrTdgXnzD61nvOs2ygvVZwp5
openUrl ctx_ver=Z39.88-2004&ctx_enc=info%3Aofi%2Fenc%3AUTF-8&rfr_id=info%3Asid%2Fsummon.serialssolutions.com&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Easily+Parallelizable+and+Distributable+Class+of+Algorithms+for+Structured+Sparsity%2C+With+Optimal+Acceleration&rft.jtitle=Journal+of+computational+and+graphical+statistics&rft.au=Ko%2C+Seyoon&rft.au=Yu%2C+Donghyeon&rft.au=Won%2C+Joong-Ho&rft.date=2019-10-02&rft.pub=American+Statistical+Association%2C+the+Institute+of+Mathematical+Statistics%2C+and+the+Interface+Foundation+of+North+America&rft.issn=1061-8600&rft.eissn=1537-2715&rft.volume=28&rft.issue=4&rft.spage=821&rft.epage=833&rft_id=info:doi/10.1080%2F10618600.2019.1592757&rft.externalDocID=45282221
thumbnail_l http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/lc.gif&issn=1061-8600&client=summon
thumbnail_m http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/mc.gif&issn=1061-8600&client=summon
thumbnail_s http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/sc.gif&issn=1061-8600&client=summon