Network-Offloaded Bandwidth-Optimal Broadcast and Allgather for Distributed AI

In the Fully Sharded Data Parallel (FSDP) training pipeline, collective operations can be interleaved to maximize the communication/computation overlap. In this scenario, outstanding operations such as Allgather and Reduce-Scatter can compete for the injection bandwidth and create pipeline bubbles....

Celý popis

Podrobná bibliografia

Vydané v:	SC24: International Conference for High Performance Computing, Networking, Storage and Analysis s. 1 - 17
Hlavní autori:	Khalilov, Mikhail, Girolamo, Salvatore Di, Chrapek, Marcin, Nudelman, Rami, Bloch, Gil, Hoefler, Torsten
Médium:	Konferenčný príspevok..
Jazyk:	English
Vydavateľské údaje:	IEEE 17.11.2024
Predmet:	AI accelerators Artificial intelligence Clusters Engines Multicast algorithms Networking Next generation networking Pipelines Protocols Substrates Supercomputers Training
On-line prístup:	Získať plný text
Tagy:	Pridať tag Žiadne tagy, Buďte prvý, kto otaguje tento záznam!