Why Globally Re-shuffle? Revisiting Data Shuffling in Large Scale Deep Learning

Stochastic gradient descent (SGD) is the most prevalent algorithm for training Deep Neural Networks (DNN). SGD iterates the input data set in each training epoch processing data samples in a random access fashion. Because this puts enormous pressure on the I/O subsystem, the most common approach to...

Full description

Saved in:
Bibliographic Details
Published in:Proceedings - IEEE International Parallel and Distributed Processing Symposium pp. 1085 - 1096
Main Authors: Nguyen, Truong Thao, Trahay, Francois, Domke, Jens, Drozd, Aleksandr, Vatai, Emil, Liao, Jianwei, Wahib, Mohamed, Gerofi, Balazs
Format: Conference Proceeding
Language:English
Published: IEEE 01.05.2022
Subjects:
Costs
Data Shuffling
Deep learning
Distributed databases
Distributed Deep Learning
Distributed processing
I/O
Neural networks
Stochastic processes
Training
ISSN:1530-2075
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:Stochastic gradient descent (SGD) is the most prevalent algorithm for training Deep Neural Networks (DNN). SGD iterates the input data set in each training epoch processing data samples in a random access fashion. Because this puts enormous pressure on the I/O subsystem, the most common approach to distributed SGD in HPC environments is to replicate the entire dataset to node local SSDs. However, due to rapidly growing data set sizes this approach has become increasingly infeasible. Surprisingly, the questions of why and to what extent random access is required have not received a lot of attention in the literature from an empirical standpoint. In this paper, we revisit data shuffling in DL workloads to investigate the viability of partitioning the dataset among workers and performing only a partial distributed exchange of samples in each training epoch. Through extensive experiments on up to 2,048 GPUs of ABCI and 4,096 compute nodes of Fugaku, we demonstrate that in practice validation accuracy of global shuffling can be maintained when carefully tuning the partial distributed exchange. We provide a solution implemented in PyTorch that enables users to control the proposed data exchange scheme.
ISSN:1530-2075
DOI:10.1109/IPDPS53621.2022.00109