An Intelligent Optimization-Based Load Balancing Scheduling Algorithm for Flink Tasks

To address the issues of resource usage imbalance and throughput bottlenecks caused by Flink's default scheduling strategy, a balanced scheduling algorithm based on Simulated Annealing and Particle Swarm Optimization (SA-PSOBS) is proposed. First, by real-time collection of task monitoring and...

Celý popis

Uloženo v:
Podrobná bibliografie
Vydáno v:2025 4th International Symposium on Computer Applications and Information Technology (ISCAIT) s. 2091 - 2096
Hlavní autoři: Pan, Yuepeng, Liu, Yong
Médium: Konferenční příspěvek
Jazyk:angličtina
Vydáno: IEEE 21.03.2025
Témata:
Adaptive dynamic scheduling algorithm
Balanced scheduling
Big data
Distributed computing
Dynamic scheduling
Heuristic algorithm
Heuristic algorithms
Load management
Optimization
Parallel processing
Particle swarm optimization
Resource management
Scheduling algorithms
Simulated annealing
Stream computing
Throughput
On-line přístup:Získat plný text
Tagy: Přidat tag
Žádné tagy, Buďte první, kdo vytvoří štítek k tomuto záznamu!
Popis
Shrnutí:To address the issues of resource usage imbalance and throughput bottlenecks caused by Flink's default scheduling strategy, a balanced scheduling algorithm based on Simulated Annealing and Particle Swarm Optimization (SA-PSOBS) is proposed. First, by real-time collection of task monitoring and load metrics, the scheduling results are dynamically adjusted, resolving the weight deficiency problem of static scheduling strategies. Second, a two-stage optimization approach is adopted: in the Task-to-Slot allocation phase, the Simulated Annealing algorithm is used to optimize resource load balancing among Slots; in the Slot-to-TaskManager allocation phase, the Particle Swarm Optimization algorithm is employed to balance resource differences among TaskManagers. Additionally, a Slots waiting mechanism is introduced, combined with weighted fitness calculations based on feature matrices of tasks, Slots, and TaskManagers, to further optimize global resource allocation. Experiments conducted on the NASA-HTTP-LOG dataset and WordCount job show that for a job with 100 tasks, throughput increases by 9.673 \% under globally consistent parallelism and by 41.929% under non-consistent parallelism; for a job with 500 tasks, throughput increases by 10.919 \% under globally consistent parallelism and by 16.631 \% under non-consistent parallelism. The proposed algorithm effectively resolves resource imbalance and throughput bottlenecks, outperforming both the default scheduling strategy and the Round-Robin static load balancing algorithm.
DOI:10.1109/ISCAIT64916.2025.11010314