Communication Algorithm-Architecture Co-Design for Distributed Deep Learning

Large-scale distributed deep learning training has enabled developments of more complex deep neural network models to learn from larger datasets for sophisticated tasks. In particular, distributed stochastic gradient descent intensively invokes all-reduce operations for gradient update, which domina...

Ausführliche Beschreibung

Bibliographische Detailangaben

Veröffentlicht in:	Proceedings - International Symposium on Computer Architecture S. 181 - 194
Hauptverfasser:	Huang, Jiayi, Majumder, Pritam, Kim, Sungkeun, Muzahid, Abdullah, Yum, Ki Hwan, Kim, Eun Jung
Format:	Tagungsbericht
Sprache:	Englisch
Veröffentlicht:	IEEE 01.06.2021
Schlagworte:	algorithm-architecture co-design all-reduce data-parallel training Deep learning distributed deep learning interconnection network Network topology Schedules Scheduling Stochastic processes Topology Training
ISSN:	2575-713X
Online-Zugang:	Volltext
Tags:	Tag hinzufügen Keine Tags, Fügen Sie den ersten Tag hinzu!