A Multidimensional Communication Scheduling Method for Hybrid Parallel DNN Training

The transformer-based deep neural network (DNN) models have shown considerable success across diverse tasks, prompting widespread adoption of distributed training methods such as data parallelism and pipeline parallelism. With the increasing parameter number, hybrid parallel training becomes imperat...

Celý popis

Uloženo v:

Podrobná bibliografie
Vydáno v:	IEEE transactions on parallel and distributed systems Ročník 35; číslo 8; s. 1415 - 1428
Hlavní autoři:	Li, Shengwei, Lu, Kai, Lai, Zhiquan, Liu, Weijie, Ge, Keshi, Li, Dongsheng
Médium:	Journal Article
Jazyk:	angličtina
Vydáno:	New York IEEE 01.08.2024 The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
Témata:	Artificial neural networks Communication communication scheduling Computation Computational modeling Data models deep learning Distributed training hybrid parallelism Parallel processing Pipeline processing Pipelines Pipelining (computers) Processor scheduling Real time Scheduling Topology Training Transformers
ISSN:	1045-9219, 1558-2183
On-line přístup:	Získat plný text
Tagy:	Přidat tag Žádné tagy, Buďte první, kdo vytvoří štítek k tomuto záznamu!

Buďte první, kdo okomentuje tento záznam!