BirdMoE: Reducing Communication Costs for Mixture-of-Experts Training Using Load-Aware Bi-random Quantization

Mixture-of-Experts (MoE) model parallelism is prevalent in training Large Language Models (e.g., ChatGPT). However, the intensive all-to-all collective communication of the MoE layer's intermediate computing results substantially degrades MoE training efficiency. In this paper, we propose BirdM...

Celý popis

Podrobná bibliografie

Vydáno v:	2025 62nd ACM/IEEE Design Automation Conference (DAC) s. 1 - 7
Hlavní autoři:	Wu, Donglei, Yang, Weihao, Zou, Xiangyu, Jia, Jinda, Tao, Dingwen, Xia, Wen, Tian, Zhihong
Médium:	Konferenční příspěvek
Jazyk:	angličtina
Vydáno:	IEEE 22.06.2025
Témata:	Chatbots communication -Mixture-of-Experts training communication compression Computational modeling Costs Design automation Large language models load balance Load modeling Mixture-of-Experts training Parallel processing Quantization (signal) Training
On-line přístup:	Získat plný text
Tagy:	Přidat tag Žádné tagy, Buďte první, kdo vytvoří štítek k tomuto záznamu!