Communication Algorithm-Architecture Co-Design for Distributed Deep Learning

Large-scale distributed deep learning training has enabled developments of more complex deep neural network models to learn from larger datasets for sophisticated tasks. In particular, distributed stochastic gradient descent intensively invokes all-reduce operations for gradient update, which domina...

Full description

Bibliographic Details

Published in:	Proceedings - International Symposium on Computer Architecture pp. 181 - 194
Main Authors:	Huang, Jiayi, Majumder, Pritam, Kim, Sungkeun, Muzahid, Abdullah, Yum, Ki Hwan, Kim, Eun Jung
Format:	Conference Proceeding
Language:	English
Published:	IEEE 01.06.2021
Subjects:	algorithm-architecture co-design all-reduce data-parallel training Deep learning distributed deep learning interconnection network Network topology Schedules Scheduling Stochastic processes Topology Training
ISSN:	2575-713X
Online Access:	Get full text
Tags:	Add Tag No Tags, Be the first to tag this record!