An Intelligent Optimization-Based Load Balancing Scheduling Algorithm for Flink Tasks - Pan, Yuepeng; Liu, Yong (2025 4th International Symposium on Computer Applications and Information Technology (ISCAIT), 2025)
To address the issues of resource usage imbalance and throughput bottlenecks caused by Flink's default scheduling strategy, a balanced scheduling algorithm based on Simulated Annealing and Particle Swarm Optimization (SA-PSOBS) is proposed. First, by real-time collection of task monitoring and load metrics, the scheduling results are dynamically adjusted, resolving the weight deficiency problem of static scheduling strategies. Second, a two-stage optimization approach is adopted: in the Task-to-Slot allocation phase, the Simulated Annealing algorithm is used to optimize resource load balancing among Slots; in the Slot-to-TaskManager allocation phase, the Particle Swarm Optimization algorithm is employed to balance resource differences among TaskManagers. Additionally, a Slots waiting mechanism is introduced, combined with weighted fitness calculations based on feature matrices of tasks, Slots, and TaskManagers, to further optimize global resource allocation. Experiments conducted on the NASA-HTTP-LOG dataset and WordCount job show that for a job with 100 tasks, throughput increases by 9.673 % under globally consistent parallelism and by 41.929% under non-consistent parallelism; for a job with 500 tasks, throughput increases by 10.919 % under globally consistent parallelism and by 16.631 % under non-consistent parallelism. The proposed algorithm effectively resolves resource imbalance and throughput bottlenecks, outperforming both the default scheduling strategy and the Round-Robin static load balancing algorithm.