Doubly random parallel stochastic methods for large scale learning

We consider learning problems over training sets in which both, the number of training examples and the dimension of the feature vectors, are large. To solve these problems we propose the random parallel stochastic algorithm (RAPSA). We call the algorithm random parallel because it utilizes multiple...

Full description

Saved in:
Bibliographic Details
Published in:Proceedings of the American Control Conference pp. 4847 - 4852
Main Authors: Mokhtari, Aryan, Koppel, Alec, Ribeiro, Alejandro
Format: Conference Proceeding Journal Article
Language:English
Published: American Automatic Control Council (AACC) 01.07.2016
Subjects:
Algorithms
Convergence
Learning
Linear programming
Mathematical analysis
Optimization
Parallel processing
Processors
Program processors
Radio frequency
Stochasticity
Training
Vectors (mathematics)
ISSN:2378-5861
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Abstract We consider learning problems over training sets in which both, the number of training examples and the dimension of the feature vectors, are large. To solve these problems we propose the random parallel stochastic algorithm (RAPSA). We call the algorithm random parallel because it utilizes multiple processors to operate in a randomly chosen subset of blocks of the feature vector. We call the algorithm parallel stochastic because processors choose elements of the training set randomly and independently. Algorithms that are parallel in either of these dimensions exist, but RAPSA is the first attempt at a methodology that is parallel in both, the selection of blocks and the selection of elements of the training set. In RAPSA, processors utilize the randomly chosen functions to compute the stochastic gradient component associated with a randomly chosen block. The technical contribution of this paper is to show that this minimally coordinated algorithm converges to the optimal classifier when the training objective is convex. In particular, we show that: (i) When using decreasing stepsizes, RAPSA converges almost surely over the random choice of blocks and functions. (ii) When using constant stepsizes, convergence is to a neighborhood of optimality with a rate that is linear in expectation. RAPSA is numerically evaluated on the MNIST digit recognition problem.
AbstractList We consider learning problems over training sets in which both, the number of training examples and the dimension of the feature vectors, are large. To solve these problems we propose the random parallel stochastic algorithm (RAPSA). We call the algorithm random parallel because it utilizes multiple processors to operate in a randomly chosen subset of blocks of the feature vector. We call the algorithm parallel stochastic because processors choose elements of the training set randomly and independently. Algorithms that are parallel in either of these dimensions exist, but RAPSA is the first attempt at a methodology that is parallel in both, the selection of blocks and the selection of elements of the training set. In RAPSA, processors utilize the randomly chosen functions to compute the stochastic gradient component associated with a randomly chosen block. The technical contribution of this paper is to show that this minimally coordinated algorithm converges to the optimal classifier when the training objective is convex. In particular, we show that: (i) When using decreasing stepsizes, RAPSA converges almost surely over the random choice of blocks and functions. (ii) When using constant stepsizes, convergence is to a neighborhood of optimality with a rate that is linear in expectation. RAPSA is numerically evaluated on the MNIST digit recognition problem.
Author Mokhtari, Aryan
Koppel, Alec
Ribeiro, Alejandro
Author_xml – sequence: 1
  givenname: Aryan
  surname: Mokhtari
  fullname: Mokhtari, Aryan
  email: aryanm@seas.upenn.edu
  organization: Dept. of Electr. & Syst. Eng., Univ. of Pennsylvania, Philadelphia, PA, USA
– sequence: 2
  givenname: Alec
  surname: Koppel
  fullname: Koppel, Alec
  email: akoppel@seas.upenn.edu
  organization: Dept. of Electr. & Syst. Eng., Univ. of Pennsylvania, Philadelphia, PA, USA
– sequence: 3
  givenname: Alejandro
  surname: Ribeiro
  fullname: Ribeiro, Alejandro
  email: aribeiro@seas.upenn.edu
  organization: Dept. of Electr. & Syst. Eng., Univ. of Pennsylvania, Philadelphia, PA, USA
BookMark eNotkD1PwzAURQ0CiaawI7F4ZEnxs-PEHkv4lCqxwBy9pM9tkBMXOx3676nUTvcMR2e4Gbsaw0iM3YNYAAj7tKzrhRRQLiotS5DigmVQlJUypZHiks2kqkyuTQk3LEvpVwiwthQz9vwS9q0_8IjjOgx8hxG9J8_TFLotpqnv-EDTNqwTdyFyj3FDPHXoiXvCOPbj5pZdO_SJ7s47Zz9vr9_1R776ev-sl6u8l8JMudSu6Cy6CjRiYQl1oZxCMIhHPHKlsOxQAICzrZRFZwrSqkWxdta1oObs8dTdxfC3pzQ1Q5868h5HCvvUgFFa28ro6qg-nNSeiJpd7AeMh-Z8jfoHt7BZdw
ContentType Conference Proceeding
Journal Article
DBID 6IE
6IH
CBEJK
RIE
RIO
7SP
8FD
L7M
DOI 10.1109/ACC.2016.7526120
DatabaseName IEEE Electronic Library (IEL) Conference Proceedings
IEEE Proceedings Order Plan (POP) 1998-present by volume
IEEE Xplore All Conference Proceedings
IEEE Electronic Library (IEL)
IEEE Proceedings Order Plans (POP) 1998-present
Electronics & Communications Abstracts
Technology Research Database
Advanced Technologies Database with Aerospace
DatabaseTitle Technology Research Database
Advanced Technologies Database with Aerospace
Electronics & Communications Abstracts
DatabaseTitleList
Technology Research Database
Database_xml – sequence: 1
  dbid: RIE
  name: IEEE Electronic Library (IEL)
  url: https://ieeexplore.ieee.org/
  sourceTypes: Publisher
DeliveryMethod fulltext_linktorsrc
Discipline Engineering
EISBN 1467386820
9781467386807
9781467386821
1467386804
EISSN 2378-5861
EndPage 4852
ExternalDocumentID 7526120
Genre orig-research
GroupedDBID -~X
23M
29O
6IE
6IF
6IH
6IK
6IL
6IM
6IN
AAJGR
AAWTH
ABLEC
ACGFS
ADZIZ
AFFNX
ALMA_UNASSIGNED_HOLDINGS
BEFXN
BFFAM
BGNUA
BKEBE
BPEOZ
CBEJK
CHZPO
IEGSK
IJVOP
IPLJI
M43
OCL
RIE
RIL
RIO
RNS
7SP
8FD
L7M
ID FETCH-LOGICAL-i208t-25f4c9af715aa49ea543f3a18aaa5443f73a6ca0111f9b224c84e53ba0df9fb13
IEDL.DBID RIE
ISICitedReferencesCount 9
ISICitedReferencesURI http://www.webofscience.com/api/gateway?GWVersion=2&SrcApp=Summon&SrcAuth=ProQuest&DestLinkType=CitingArticles&DestApp=WOS_CPL&KeyUT=000388376104152&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D
IngestDate Thu Jul 10 23:39:14 EDT 2025
Wed Aug 27 02:11:24 EDT 2025
IsPeerReviewed false
IsScholarly false
Language English
LinkModel DirectLink
MergedId FETCHMERGED-LOGICAL-i208t-25f4c9af715aa49ea543f3a18aaa5443f73a6ca0111f9b224c84e53ba0df9fb13
Notes ObjectType-Article-2
SourceType-Scholarly Journals-1
ObjectType-Conference-1
ObjectType-Feature-3
content type line 23
SourceType-Conference Papers & Proceedings-2
PQID 1835597857
PQPubID 23500
PageCount 6
ParticipantIDs proquest_miscellaneous_1835597857
ieee_primary_7526120
PublicationCentury 2000
PublicationDate 20160701
PublicationDateYYYYMMDD 2016-07-01
PublicationDate_xml – month: 07
  year: 2016
  text: 20160701
  day: 01
PublicationDecade 2010
PublicationTitle Proceedings of the American Control Conference
PublicationTitleAbbrev ACC
PublicationYear 2016
Publisher American Automatic Control Council (AACC)
Publisher_xml – name: American Automatic Control Council (AACC)
SSID ssj0019960
Score 1.6796154
Snippet We consider learning problems over training sets in which both, the number of training examples and the dimension of the feature vectors, are large. To solve...
SourceID proquest
ieee
SourceType Aggregation Database
Publisher
StartPage 4847
SubjectTerms Algorithms
Convergence
Learning
Linear programming
Mathematical analysis
Optimization
Parallel processing
Processors
Program processors
Radio frequency
Stochasticity
Training
Vectors (mathematics)
Title Doubly random parallel stochastic methods for large scale learning
URI https://ieeexplore.ieee.org/document/7526120
https://www.proquest.com/docview/1835597857
WOSCitedRecordID wos000388376104152&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D
hasFullText 1
inHoldings 1
isFullTextHit
isPrint
link http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV09T8MwELVKxQALHy2ifMlIjKRNGju2R6iomKoOIHWLzs4ZKoUUNS0S_x47SQsSLGzOYCm6s3zvfHfvEXKDTGqbJBgkQpuAoVtJLt1nZBx-BQ-Za7EJMZnI2UxNW-R2OwuDiFXzGfb9sqrlZwuz9k9lA8E94ZVL0HeESOpZrW3FwLOMbMqQoRrcjUa-byvpN3sa8ZRfN24VRsYH__uBQ9L9nsej022kOSItLI7J_g8qwQ65d0hY55_UhZ5s8UY9o3eeY04dtjOv4MmYaS0WXVIHU2nuG8Bp6RyEtBGOeOmS5_HD0-gxaPQRgvkwlKtgyC0zCqyIOABTCJzFNoZIAoCntbMihsSAV5O3SrtYbSRDHmsIM6usjuIT0i4WBZ4SGgI3GnQsLXcJiciACSXjxDLNTahC7JGOt0b6XlNgpI0heuR6Y87UHUtfa4ACF-sydTeFz1UkF2d_bz0ne94_defrBWmvlmu8JLvmYzUvl1eVb78AimmmZg
linkProvider IEEE
linkToHtml http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV3PS8MwFA5DBfXij02cPyN4tFu7Jk1y1OGYOMcOE3YrL1mig9rKugn-9yZtNwW9eEsPgfJeyPte3nvfh9C1JlyaKNJexKTyiLYrTrn9DJTFr-Agcyk2wYZDPpmIUQ3drGdhtNZF85luuWVRy59maumeytqMOsIrm6BvUkI6fjmtta4ZOJ6RVSHSF-3bbtd1bkWtalcln_Lrzi0CSW_vf7-wjxrfE3l4tI41B6im00O0-4NMsI7uLBaWySe2wWeavWHH6Z0kOsEW3alXcHTMuJSLzrEFqjhxLeA4ty7SuJKOeGmg5979uNv3KoUEb9bx-cLrUEOUAMMCCkCEBkpCE0LAAcAR2xkWQqTA6ckbIW20VpxoGkrwp0YYGYRHaCPNUn2MsA9USZAhN9SmJGwKhAkeRoZIqnzh6yaqO2vE7yUJRlwZoomuVuaM7cF01QZIdbbMY3tXuGyFU3by99ZLtN0fPw3iwcPw8RTtOF-VfbBnaGMxX-pztKU-FrN8flH4-Qtedqmt
openUrl ctx_ver=Z39.88-2004&ctx_enc=info%3Aofi%2Fenc%3AUTF-8&rfr_id=info%3Asid%2Fsummon.serialssolutions.com&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=proceeding&rft.title=Proceedings+of+the+American+Control+Conference&rft.atitle=Doubly+random+parallel+stochastic+methods+for+large+scale+learning&rft.au=Mokhtari%2C+Aryan&rft.au=Koppel%2C+Alec&rft.au=Ribeiro%2C+Alejandro&rft.date=2016-07-01&rft.pub=American+Automatic+Control+Council+%28AACC%29&rft.eissn=2378-5861&rft.spage=4847&rft.epage=4852&rft_id=info:doi/10.1109%2FACC.2016.7526120&rft.externalDocID=7526120