Zobraziť v EDS

Interval-Asynchrony: Delimited Intervals of Localised Asynchrony for Fast Parallel SGD

Uložené v:
Podrobná bibliografia
Názov: Interval-Asynchrony: Delimited Intervals of Localised Asynchrony for Fast Parallel SGD
Autori: Garby, Jacob Stacey, 2001, Tsigas, Philippas, 1967
Zdroj: Relaxed Semantics Across the Data Analytics Stack (RELAX-DN) 31st International Conference on Parallel and Distributed Computing, Euro-Par 2025, Dresden, Germany Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) Artifact of the paper: Interval-Asynchrony: Delimited Intervals of Localised Asynchrony for Fast Parallel SGD. 15901 LNCS:236-249
Predmety: Parallel SGD, Asynchronous Data Processing, Staleness, Parallel Algorithms
Popis: Stochastic gradient descent (SGD) is a crucial optimisation algorithm due to its ubiquity in machine learning applications. Parallelism is a popular approach to scale SGD, but the standard synchronous formulation struggles due to significant synchronisation overhead. For this reason, asynchronous implementations are increasingly common. These provide an improvement in throughput at the expense of introducing stale gradients which reduce model accuracy. Previous approaches to mitigate the downsides of asynchronous processing include adaptively adjusting the number of worker threads or the learning rate, but at their core these are still fully asynchronous and hence still suffer from lower accuracy due to more staleness. We propose Interval-Asynchrony, a semi-asynchronous method which retains high throughput while reducing gradient staleness, both on average as well as with a hard upper bound. Our method achieves this by introducing periodic asynchronous intervals, within which SGD is executed asynchronously, but between which gradient computations may not cross. The size of these intervals determines the degree of asynchrony, providing us with an adjustable scale. Since the optimal interval size varies over time, we additionally provide two strategies for dynamic adjustment thereof. We evaluate our method against several baselines on the CIFAR-10 and CIFAR-100 datasets, and demonstrate a 32% decrease in training time as well as improved scalability up to 128 threads.
Prístupová URL adresa: https://research.chalmers.se/publication/548121
Databáza: SwePub
FullText Text:
  Availability: 0
CustomLinks:
  – Url: https://www.webofscience.com/api/gateway?GWVersion=2&SrcApp=EBSCO&SrcAuth=EBSCO&DestApp=WOS&ServiceName=TransferToWoS&DestLinkType=GeneralSearchSummary&Func=Links&author=Garby%20JS
    Name: ISI
    Category: fullText
    Text: Nájsť tento článok vo Web of Science
    Icon: https://imagesrvr.epnet.com/ls/20docs.gif
    MouseOverText: Nájsť tento článok vo Web of Science
Header DbId: edsswe
DbLabel: SwePub
An: edsswe.oai.research.chalmers.se.e7b88135.ff05.487f.99b0.da722c44dc3a
RelevancyScore: 981
AccessLevel: 6
PubType: Conference
PubTypeId: conference
PreciseRelevancyScore: 981.151794433594
IllustrationInfo
Items – Name: Title
  Label: Title
  Group: Ti
  Data: Interval-Asynchrony: Delimited Intervals of Localised Asynchrony for Fast Parallel SGD
– Name: Author
  Label: Authors
  Group: Au
  Data: <searchLink fieldCode="AR" term="%22Garby%2C+Jacob+Stacey%22">Garby, Jacob Stacey</searchLink>, 2001<br /><searchLink fieldCode="AR" term="%22Tsigas%2C+Philippas%22">Tsigas, Philippas</searchLink>, 1967
– Name: TitleSource
  Label: Source
  Group: Src
  Data: <i>Relaxed Semantics Across the Data Analytics Stack (RELAX-DN) 31st International Conference on Parallel and Distributed Computing, Euro-Par 2025, Dresden, Germany Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) Artifact of the paper: Interval-Asynchrony: Delimited Intervals of Localised Asynchrony for Fast Parallel SGD</i>. 15901 LNCS:236-249
– Name: Subject
  Label: Subject Terms
  Group: Su
  Data: <searchLink fieldCode="DE" term="%22Parallel+SGD%22">Parallel SGD</searchLink><br /><searchLink fieldCode="DE" term="%22Asynchronous+Data+Processing%22">Asynchronous Data Processing</searchLink><br /><searchLink fieldCode="DE" term="%22Staleness%22">Staleness</searchLink><br /><searchLink fieldCode="DE" term="%22Parallel+Algorithms%22">Parallel Algorithms</searchLink>
– Name: Abstract
  Label: Description
  Group: Ab
  Data: Stochastic gradient descent (SGD) is a crucial optimisation algorithm due to its ubiquity in machine learning applications. Parallelism is a popular approach to scale SGD, but the standard synchronous formulation struggles due to significant synchronisation overhead. For this reason, asynchronous implementations are increasingly common. These provide an improvement in throughput at the expense of introducing stale gradients which reduce model accuracy. Previous approaches to mitigate the downsides of asynchronous processing include adaptively adjusting the number of worker threads or the learning rate, but at their core these are still fully asynchronous and hence still suffer from lower accuracy due to more staleness. We propose Interval-Asynchrony, a semi-asynchronous method which retains high throughput while reducing gradient staleness, both on average as well as with a hard upper bound. Our method achieves this by introducing periodic asynchronous intervals, within which SGD is executed asynchronously, but between which gradient computations may not cross. The size of these intervals determines the degree of asynchrony, providing us with an adjustable scale. Since the optimal interval size varies over time, we additionally provide two strategies for dynamic adjustment thereof. We evaluate our method against several baselines on the CIFAR-10 and CIFAR-100 datasets, and demonstrate a 32% decrease in training time as well as improved scalability up to 128 threads.
– Name: URL
  Label: Access URL
  Group: URL
  Data: <link linkTarget="URL" linkTerm="https://research.chalmers.se/publication/548121" linkWindow="_blank">https://research.chalmers.se/publication/548121</link>
PLink https://erproxy.cvtisr.sk/sfx/access?url=https://search.ebscohost.com/login.aspx?direct=true&site=eds-live&db=edsswe&AN=edsswe.oai.research.chalmers.se.e7b88135.ff05.487f.99b0.da722c44dc3a
RecordInfo BibRecord:
  BibEntity:
    Identifiers:
      – Type: doi
        Value: 10.1007/978-3-031-99857-7_17
    Languages:
      – Text: English
    PhysicalDescription:
      Pagination:
        PageCount: 14
        StartPage: 236
    Subjects:
      – SubjectFull: Parallel SGD
        Type: general
      – SubjectFull: Asynchronous Data Processing
        Type: general
      – SubjectFull: Staleness
        Type: general
      – SubjectFull: Parallel Algorithms
        Type: general
    Titles:
      – TitleFull: Interval-Asynchrony: Delimited Intervals of Localised Asynchrony for Fast Parallel SGD
        Type: main
  BibRelationships:
    HasContributorRelationships:
      – PersonEntity:
          Name:
            NameFull: Garby, Jacob Stacey
      – PersonEntity:
          Name:
            NameFull: Tsigas, Philippas
    IsPartOfRelationships:
      – BibEntity:
          Dates:
            – D: 01
              M: 01
              Type: published
              Y: 2026
          Identifiers:
            – Type: issn-print
              Value: 16113349
            – Type: issn-print
              Value: 03029743
            – Type: issn-locals
              Value: CTH_SWEPUB
          Numbering:
            – Type: volume
              Value: 15901 LNCS
          Titles:
            – TitleFull: Relaxed Semantics Across the Data Analytics Stack (RELAX-DN) 31st International Conference on Parallel and Distributed Computing, Euro-Par 2025, Dresden, Germany Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) Artifact of the paper: Interval-Asynchrony: Delimited Intervals of Localised Asynchrony for Fast Parallel SGD
              Type: main
ResultId 1