Merak: An Efficient Distributed DNN Training Framework with Automated 3D Parallelism for Giant Foundation Models

Foundation models are in the process of becoming the dominant deep learning technology. Pretraining a foundation model is always time-consuming due to the large scale of both the model parameter and training dataset. Besides being computing-intensive, the pretraining process is extremely memory- and...

Celý popis

Uloženo v:

Podrobná bibliografie
Vydáno v:	IEEE transactions on parallel and distributed systems Ročník 34; číslo 5; s. 1 - 13
Hlavní autoři:	Lai, Zhiquan, Li, Shengwei, Tang, Xudong, Ge, Keshi, Liu, Weijie, Duan, Yabo, Qiao, Linbo, Li, Dongsheng
Médium:	Journal Article
Jazyk:	angličtina
Vydáno:	New York IEEE 01.05.2023 The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
Témata:	Algorithms Automation Computation Computational modeling Computer memory Critical path Data models Deep learning Distributed Systems Foundation model training Mathematical models Parallel processing Parameters Pipelines Pipelining (computers) Resource utilization Solid modeling Tensors Three-dimensional displays Training
ISSN:	1045-9219, 1558-2183
On-line přístup:	Získat plný text
Tagy:	Přidat tag Žádné tagy, Buďte první, kdo vytvoří štítek k tomuto záznamu!

Buďte první, kdo okomentuje tento záznam!