A hard real-time scheduler for Spark on YARN

Apache Spark is a fast and general engine for large-scale data processing using distributed memory. It provides different deploy modes to meet the needs of different users and Spark on YARN is the most popular deploy mode. Different deploy modes have different scheduling mechanisms. Spark on YARN ha...

Full description

Saved in:
Bibliographic Details
Published in:2018 18th IEEE ACM International Symposium on Cluster, Cloud and Grid Computing (CCGRID) pp. 645 - 652
Main Authors: Wang, Guolu, Xu, Jungang, Liu, Renfeng, Huang, Shanshan
Format: Conference Proceeding
Language:English
Published: Piscataway, NJ, USA IEEE Press 01.05.2018
IEEE
Series:ACM Conferences
Subjects:
Big Data
deadline
hard real-time
Processor scheduling
Real-time systems
Scheduling
Spark
Sparks
Task analysis
value density
Yarn
hard real-time
Spark
value density
deadline
YARN
ISBN:1538658151, 9781538658154
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:Apache Spark is a fast and general engine for large-scale data processing using distributed memory. It provides different deploy modes to meet the needs of different users and Spark on YARN is the most popular deploy mode. Different deploy modes have different scheduling mechanisms. Spark on YARN has three different schedulers, including FIFO Scheduler, Fair Scheduler, and Capacity Scheduler. However, these three schedulers cannot fit hard real-time application scenarios. With the application of Apache Spark more widely, the needs of hard real-time scheduling will increase quickly. In this paper, we proposed a novel hard realtime scheduling algorithm called DVDA (Deadline and Value Density-Aware) in order to meet the requirements of hard realtime scheduling. Compared with traditional EDF (Earliest Deadline First) algorithm which only considers the deadline, the DVDA algorithm considers both the deadline and value density of the application. Furthermore, we implement a DVDA Scheduler for Spark on YARN based on the DVDA algorithm. Finally, the experiments are conducted to verify the effectiveness of the algorithm. Experimental results show that the proposed algorithm can increase the application completed rate by 18% and 6%, Value Income by 78% and 32% compared with default Capacity scheduler and EDF-Capacity scheduler respectively.
ISBN:1538658151
9781538658154
DOI:10.1109/CCGRID.2018.00096