Dynamic energy efficient data placement and cluster reconfiguration algorithm for MapReduce framework

With the recent emergence of cloud computing based services on the Internet, MapReduce and distributed file systems like HDFS have emerged as the paradigm of choice for developing large scale data intensive applications. Given the scale at which these applications are deployed, minimizing power cons...

Full description

Saved in:
Bibliographic Details
Published in:Future generation computer systems Vol. 28; no. 1; pp. 119 - 127
Main Authors: Maheshwari, Nitesh, Nanduri, Radheshyam, Varma, Vasudeva
Format: Journal Article
Language:English
Published: Elsevier B.V 2012
Subjects:
Cloud computing
Energy efficiency
MapReduce
Cloud computing
Energy efficiency
MapReduce
ISSN:0167-739X, 1872-7115
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:With the recent emergence of cloud computing based services on the Internet, MapReduce and distributed file systems like HDFS have emerged as the paradigm of choice for developing large scale data intensive applications. Given the scale at which these applications are deployed, minimizing power consumption of these clusters can significantly cut down operational costs and reduce their carbon footprint—thereby increasing the utility from a provider’s point of view. This paper addresses energy conservation for clusters of nodes that run MapReduce jobs. The algorithm dynamically reconfigures the cluster based on the current workload and turns cluster nodes on or off when the average cluster utilization rises above or falls below administrator specified thresholds, respectively. We evaluate our algorithm using the GridSim toolkit and our results show that the proposed algorithm achieves an energy reduction of 33% under average workloads and up to 54% under low workloads. ► Addressed the problem of energy conservation for large datacenters that run MapReduce jobs. ► Proposed an energy efficient data placement and a cluster reconfiguration algorithm. ► Dynamically scale the cluster in accordance with the workload imposed on it. ► The results show energy savings of 54% under low workloads and 33% under average workloads.
ISSN:0167-739X
1872-7115
DOI:10.1016/j.future.2011.07.001