BirdMoE: Reducing Communication Costs for Mixture-of-Experts Training Using Load-Aware Bi-random Quantization

Mixture-of-Experts (MoE) model parallelism is prevalent in training Large Language Models (e.g., ChatGPT). However, the intensive all-to-all collective communication of the MoE layer's intermediate computing results substantially degrades MoE training efficiency. In this paper, we propose BirdM...

Full description

Saved in:
Bibliographic Details
Published in:2025 62nd ACM/IEEE Design Automation Conference (DAC) pp. 1 - 7
Main Authors: Wu, Donglei, Yang, Weihao, Zou, Xiangyu, Jia, Jinda, Tao, Dingwen, Xia, Wen, Tian, Zhihong
Format: Conference Proceeding
Language:English
Published: IEEE 22.06.2025
Subjects:
Chatbots
communication -Mixture-of-Experts training
communication compression
Computational modeling
Costs
Design automation
Large language models
load balance
Load modeling
Mixture-of-Experts training
Parallel processing
Quantization (signal)
Training
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:Mixture-of-Experts (MoE) model parallelism is prevalent in training Large Language Models (e.g., ChatGPT). However, the intensive all-to-all collective communication of the MoE layer's intermediate computing results substantially degrades MoE training efficiency. In this paper, we propose BirdMoE, a novel load-aware communication compression technique with Bi-random quantization for MoE training with two core modules. Specifically, BirdMoE employs a lightweight Random Quantization (RQ) with expectation invariance property to efficiently map the floating-point intermediate computing results into integers while maintaining the MoE training quality. Additionally, BirdMoE utilizes a Mixed Precision (MP) strategy to dynamically balance the communication loads among expert nodes, significantly improving all-to-all communication efficiency for the MoE training system. Experiments on four typical MoE training tasks demonstrate that BirdMoE achieves higher 4.06 \times- 10.44 \times total communication compression ratios and 1.18 \times-5.27 \times training speedup compared with the state-of-the-art compression techniques while maintaining the MoE training quality.
DOI:10.1109/DAC63849.2025.11132853