Map-Balance-Reduce: An improved parallel programming model for load balancing of MapReduce

With the advent of the era of big data, the demand of massive data processing applications is also growing. Currently, MapReduce is the most commonly used data processing programming model. However, in some data processing cases, it has some defects. MapReduce programming based on key/value pairs, m...

Ausführliche Beschreibung

Gespeichert in:
Bibliographische Detailangaben
Veröffentlicht in:Future generation computer systems Jg. 105; S. 993 - 1001
Hauptverfasser: Li, Jianjiang, Liu, Yajun, Pan, Jian, Zhang, Peng, Chen, Wei, Wang, Lizhe
Format: Journal Article
Sprache:Englisch
Veröffentlicht: Elsevier B.V 01.04.2020
Schlagworte:
Distributed computing
Load balancing
MapReduce
Scheduling
Scheduling
Load balancing
Distributed computing
MapReduce
ISSN:0167-739X
Online-Zugang:Volltext
Tags: Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
Beschreibung
Zusammenfassung:With the advent of the era of big data, the demand of massive data processing applications is also growing. Currently, MapReduce is the most commonly used data processing programming model. However, in some data processing cases, it has some defects. MapReduce programming based on key/value pairs, matches the output of the Map tasks that will be transported to Reduce task nodes. The data with same key can only be processed by a Reduce node. If the data corresponding to a particular key or several keys accounts for most of all data, the Reduce node task will generate unbalanced load. In view of this defect, this paper proposes a new parallel programming model—Map-Balance-Reduce (MBR) programming model. It runs on our improved Hadoop framework and can effectively process the special data with unbalanced keys. This paper is based on two different scheduling, the processing and self-adaption scheduling. These two scheduling are designed to achieve MBR programming model. The actual testing results show that compared with MapReduce programming model, the MBR programming model under Hadoop can achieve the improvement of 9.7% to 17.6% in efficiency when testing data distributes unevenly. Furthermore, when testing conventional even-distributed data, it will only bring 1.02% time cost. •A new programming of “Map-Balance-Reduce” is proposed.•New scheduling algorithms of preprocessing scheduling and self-adaption scheduling are designed.•The proposed work can efficiently process unevenly distributed data for MapReduce.
ISSN:0167-739X
DOI:10.1016/j.future.2017.03.013