A round robin with multiple feedback job scheduler in Hadoop

Hadoop is a distributed software platform for processing big data on a large cluster, which implements core mechanism of Google's GFS and MapReduce. The MapReduce job scheduling algorithm is one of the core technologies of Hadoop. The default job scheduler of Hadoop is FIFO, which will start th...

Full description

Saved in:
Bibliographic Details
Published in:2014 IEEE International Conference on Progress in Informatics and Computing pp. 471 - 475
Main Authors: Yintian Wang, Ruonan Rao, Yinglin Wang
Format: Conference Proceeding
Language:English
Published: IEEE 01.05.2014
Subjects:
Benchmark testing
Educational institutions
Hadoop
Job Schedule
MapReduce
Round robin
round-robin with multiple feedback
Throughput
Time factors
ISBN:9781479920334, 1479920339
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:Hadoop is a distributed software platform for processing big data on a large cluster, which implements core mechanism of Google's GFS and MapReduce. The MapReduce job scheduling algorithm is one of the core technologies of Hadoop. The default job scheduler of Hadoop is FIFO, which will start the job in the order as it is submitted, and this causes the job to be started later when it is submitted later. This paper uses the round robin with a multiple feedback algorithm to solve this problem. With this scheduler, the job which is submitted late, will get quick response and be started without long delay. And the results of experiments on the Hadoop benchmark GridMix indicate that this algorithm can reduce the average response time by 10%-50%.
ISBN:9781479920334
1479920339
DOI:10.1109/PIC.2014.6972380