Understanding and optimizing asynchronous low-precision stochastic gradient descent

Stochastic gradient descent (SGD) is one of the most popular numerical algorithms used in machine learning and other domains. Since this is likely to continue for the foreseeable future, it is important to study techniques that can make it run fast on parallel hardware. In this paper, we provide the...

Celý popis

Uloženo v:
Podrobná bibliografie
Vydáno v:2017 ACM/IEEE 44th Annual International Symposium on Computer Architecture (ISCA) s. 561 - 574
Hlavní autoři: De Sa, Christopher, Feldman, Matthew, Re, Christopher, Olukotun, Kunle
Médium: Konferenční příspěvek
Jazyk:angličtina
Vydáno: ACM 01.06.2017
Témata:
Algorithm design and analysis
asynchrony
Computational modeling
FPGA
Hardware
Logistics
low precision
multicore
Optimization
Stochastic gradient descent
Stochastic processes
Throughput
On-line přístup:Získat plný text
Tagy: Přidat tag
Žádné tagy, Buďte první, kdo vytvoří štítek k tomuto záznamu!
Popis
Shrnutí:Stochastic gradient descent (SGD) is one of the most popular numerical algorithms used in machine learning and other domains. Since this is likely to continue for the foreseeable future, it is important to study techniques that can make it run fast on parallel hardware. In this paper, we provide the first analysis of a technique called BUCKWILD! that uses both asynchronous execution and low-precision computation. We introduce the DMGC model, the first conceptualization of the parameter space that exists when implementing low-precision SGD, and show that it provides a way to both classify these algorithms and model their performance. We leverage this insight to propose and analyze techniques to improve the speed of low-precision SGD. First, we propose software optimizations that can increase throughput on existing CPUs by up to 11×. Second, we propose architectural changes, including a new cache technique we call an obstinate cache, that increase throughput beyond the limits of current-generation hardware. We also implement and analyze low-precision SGD on the FPGA, which is a promising alternative to the CPU for future SGD systems.
DOI:10.1145/3079856.3080248