Network-Offloaded Bandwidth-Optimal Broadcast and Allgather for Distributed AI

In the Fully Sharded Data Parallel (FSDP) training pipeline, collective operations can be interleaved to maximize the communication/computation overlap. In this scenario, outstanding operations such as Allgather and Reduce-Scatter can compete for the injection bandwidth and create pipeline bubbles....

Full description

Bibliographic Details

Published in:	SC24: International Conference for High Performance Computing, Networking, Storage and Analysis pp. 1 - 17
Main Authors:	Khalilov, Mikhail, Girolamo, Salvatore Di, Chrapek, Marcin, Nudelman, Rami, Bloch, Gil, Hoefler, Torsten
Format:	Conference Proceeding
Language:	English
Published:	IEEE 17.11.2024
Subjects:	AI accelerators Artificial intelligence Clusters Engines Multicast algorithms Networking Next generation networking Pipelines Protocols Substrates Supercomputers Training
Online Access:	Get full text
Tags:	Add Tag No Tags, Be the first to tag this record!