MF-Storm: a maximum flow-based job scheduler for stream processing engines on computational clusters to increase throughput

A scheduling algorithm tries to schedule multiple computational tasks on a cluster of multiple computing nodes to maximize throughput with optimal utilization of computational and communicational resources. A Stream Processing Engine (SPE) is deployed to run streaming applications (computational tas...

Celý popis

Uložené v:
Podrobná bibliografia
Vydané v:PeerJ. Computer science Ročník 8; s. e1077
Hlavní autori: Muhammad, Asif, Abdul Qadir, Muhammad
Médium: Journal Article
Jazyk:English
Vydavateľské údaje: San Diego PeerJ. Ltd 26.09.2022
PeerJ, Inc
PeerJ Inc
Predmet:
Algorithms
Algorithms and Analysis of Algorithms
APACHE storm
Artificial Intelligence
Bandwidths
Clusters
Communication
Computation
Distributed and Parallel Computing
Experiments
Heterogeneous cluster
Job scheduler
Literature reviews
Mutual funds
Network latency
Nodes
Optimization
Resource utilization
Resource-aware
Schedules
Scheduling
Social networks
Software
Stream processing engines
Task scheduling
Topology
Workloads
ISSN:2376-5992, 2376-5992
On-line prístup:Získať plný text
Tagy: Pridať tag
Žiadne tagy, Buďte prvý, kto otaguje tento záznam!
Popis
Shrnutí:A scheduling algorithm tries to schedule multiple computational tasks on a cluster of multiple computing nodes to maximize throughput with optimal utilization of computational and communicational resources. A Stream Processing Engine (SPE) is deployed to run streaming applications (computational tasks) on a computational cluster which helps execution and coordination of these applications. It is observed that there is a gap in the optimal mapping of a computational and communicational load of a streaming application on the underlying computational and communication power of the resources (cluster). Frequently communicated tasks are scheduled at different processing nodes with relatively slow communicating links. This increases network latency with a decrease in resource utilization. Hence, reduces the achieved throughput of the cluster significantly. MF-Storm, a max-flow min-cut based job scheduler is presented to achieve a near-optimum schedule to maximize throughput. It schedules a streaming application by considering the processing, communication demands, available computational and communicational resources in a heterogeneous cluster, dynamically with minimized scheduling cost. To keep the scheduling cost minimum, the scheduler is built in a pipeline with two major stages: in the first stage, the application's tasks graph is partitioned using the max-flow min-cut algorithm to minimize inter-partition traffic, and in the second stage, these partitions are assigned to computing nodes according to the computational power of the cluster's nodes. Extensive experiments were done to evaluate the performance of MF-Storm using different topologies with multiple scenarios on a physical cluster implementation. Results showed on average 148% improvement in throughput with 30% less computational resources as compared to different state-of-the-art schedulers.
Bibliografia:ObjectType-Article-1
SourceType-Scholarly Journals-1
ObjectType-Feature-2
content type line 14
content type line 23
ISSN:2376-5992
2376-5992
DOI:10.7717/peerj-cs.1077