Cross-Search With Improved Multi-Dimensional Dichotomy-Based Joint Optimization for Distributed Parallel Training of DNN

Distributed parallel training of large-scale deep neural networks (DNN) has attracted the attentions of both artificial intelligence and high-performance distributed computing. One of efficient approaches is the micro-batch-based pipeline parallelism (MBPP), e.g., GPipe and Terapipe. Based on the MB...

Full description

Saved in:

Bibliographic Details
Published in:	IEEE transactions on parallel and distributed systems Vol. 36; no. 8; pp. 1680 - 1694
Main Authors:	Zhou, Guangyao, Fu, Yiqin, Lan, Haocheng, Xie, Yuanlun, Tian, Wenhong, Buyya, Rajkumar, Qian, Jiahong, Su, Teng
Format:	Journal Article
Language:	English
Published:	IEEE 01.08.2025
Subjects:	Computational modeling Costs cross-search Data models Distributed parallelism improved multi-dimensional dichotomy large DNN Mathematical models micro-batch pipeline Optimization Parallel processing Partitioning algorithms Pipelines Training transformer Transformers
ISSN:	1045-9219, 1558-2183
Online Access:	Get full text
Tags:	Add Tag No Tags, Be the first to tag this record!

Be the first to leave a comment!