An Intelligent Optimization-Based Load Balancing Scheduling Algorithm for Flink Tasks

To address the issues of resource usage imbalance and throughput bottlenecks caused by Flink's default scheduling strategy, a balanced scheduling algorithm based on Simulated Annealing and Particle Swarm Optimization (SA-PSOBS) is proposed. First, by real-time collection of task monitoring and...

Full description

Saved in:
Bibliographic Details
Published in:2025 4th International Symposium on Computer Applications and Information Technology (ISCAIT) pp. 2091 - 2096
Main Authors: Pan, Yuepeng, Liu, Yong
Format: Conference Proceeding
Language:English
Published: IEEE 21.03.2025
Subjects:
Adaptive dynamic scheduling algorithm
Balanced scheduling
Big data
Distributed computing
Dynamic scheduling
Heuristic algorithm
Heuristic algorithms
Load management
Optimization
Parallel processing
Particle swarm optimization
Resource management
Scheduling algorithms
Simulated annealing
Stream computing
Throughput
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:To address the issues of resource usage imbalance and throughput bottlenecks caused by Flink's default scheduling strategy, a balanced scheduling algorithm based on Simulated Annealing and Particle Swarm Optimization (SA-PSOBS) is proposed. First, by real-time collection of task monitoring and load metrics, the scheduling results are dynamically adjusted, resolving the weight deficiency problem of static scheduling strategies. Second, a two-stage optimization approach is adopted: in the Task-to-Slot allocation phase, the Simulated Annealing algorithm is used to optimize resource load balancing among Slots; in the Slot-to-TaskManager allocation phase, the Particle Swarm Optimization algorithm is employed to balance resource differences among TaskManagers. Additionally, a Slots waiting mechanism is introduced, combined with weighted fitness calculations based on feature matrices of tasks, Slots, and TaskManagers, to further optimize global resource allocation. Experiments conducted on the NASA-HTTP-LOG dataset and WordCount job show that for a job with 100 tasks, throughput increases by 9.673 \% under globally consistent parallelism and by 41.929% under non-consistent parallelism; for a job with 500 tasks, throughput increases by 10.919 \% under globally consistent parallelism and by 16.631 \% under non-consistent parallelism. The proposed algorithm effectively resolves resource imbalance and throughput bottlenecks, outperforming both the default scheduling strategy and the Round-Robin static load balancing algorithm.
DOI:10.1109/ISCAIT64916.2025.11010314