SMSG: Profiling-Free Parallelism Modeling for Distributed Training of DNN

The increasing size of deep neural networks (DNNs) raises a high demand for distributed training. An expert could find good hybrid parallelism strategies, but designing suitable strategies is time and labor-consuming. Therefore, automating parallelism strategy generation is crucial and desirable for...

Full description

Saved in:

Bibliographic Details
Published in:	International journal of parallel programming Vol. 51; no. 2-3; pp. 109 - 127
Main Authors:	Wang, Haoran, Tachon, Thibaut, Li, Chong, Robert, Sophie, Limet, Sébastien
Format:	Journal Article
Language:	English
Published:	New York Springer US 01.06.2023 Springer Nature B.V Springer Verlag
Subjects:	Artificial neural networks Communication Computation Computer Science Cost analysis Cost function Homomorphisms Mathematical analysis Parallel processing Portability Processor Architectures Searching Semantics Software Engineering/Programming and Operating Systems Special Issue on High-Level Parallel Programming and Applications (HLPP 2022) Theory of Computation Training Parallel modeling Distributed training Deep network networks Performance analysis Functional transformation Symbolic cost model Distributed training ; Deep network networks; Parallel modeling; Symbolic cost model; Functional transformation; Performance analysis
ISSN:	0885-7458, 1573-7640
Online Access:	Get full text
Tags:	Add Tag No Tags, Be the first to tag this record!

Be the first to leave a comment!