Local and Global Optimization of MapReduce Program Model

MapReduce, which was introduced by Google, provides two functional interfaces, Map and Reduce, for a user to write the user-specific code to process the large amount of data. It has been widely deployed in cloud computing systems. The parallel tasks, data partition, and data transit are automaticall...

Celý popis

Uloženo v:
Podrobná bibliografie
Vydáno v:2011 IEEE World Congress on Services s. 257 - 264
Hlavní autoři: Congchong Liu, Shujia Zhou
Médium: Konferenční příspěvek
Jazyk:angličtina
Vydáno: IEEE 01.07.2011
Témata:
Adaptation models
Computational modeling
Computers
Distributed databases
distributed memory system
load balancing
Load modeling
MapReduce
Monitoring
Runtime
shared-memory system
X10 parallel programming language
ISBN:1457708795, 9781457708794
ISSN:2378-3818
On-line přístup:Získat plný text
Tagy: Přidat tag
Žádné tagy, Buďte první, kdo vytvoří štítek k tomuto záznamu!
Popis
Shrnutí:MapReduce, which was introduced by Google, provides two functional interfaces, Map and Reduce, for a user to write the user-specific code to process the large amount of data. It has been widely deployed in cloud computing systems. The parallel tasks, data partition, and data transit are automatically managed by its runtime system. This paper proposes a solution to optimize the MapReduce program model and demonstrate it with X10. We develop an adaptive load distribution scheme to balance the load on each node and consequently reduce across-node communication cost occurring in the Reduce function. In addition, we exploit shared-memory in each node to further reduce the communication cost with multi-core programming.
ISBN:1457708795
9781457708794
ISSN:2378-3818
DOI:10.1109/SERVICES.2011.64