Evaluating Hadoop for Data-Intensive Scientific Operations

Emerging sensor networks, more capable instruments, and ever increasing simulation scales are generating data at a rate that exceeds our ability to effectively manage, curate, analyze, and share it. Data-intensive computing is expected to revolutionize the next-generation software stack. Hadoop, an...

Celý popis

Uloženo v:
Podrobná bibliografie
Vydáno v:2012 IEEE Fifth International Conference on Cloud Computing s. 67 - 74
Hlavní autoři: Fadika, Z., Govindaraju, M., Canon, R., Ramakrishnan, L.
Médium: Konferenční příspěvek
Jazyk:angličtina
Vydáno: IEEE 01.06.2012
Témata:
Cloud computing
Conferences
ISBN:9781467328920, 1467328928
ISSN:2159-6182
On-line přístup:Získat plný text
Tagy: Přidat tag
Žádné tagy, Buďte první, kdo vytvoří štítek k tomuto záznamu!
Popis
Shrnutí:Emerging sensor networks, more capable instruments, and ever increasing simulation scales are generating data at a rate that exceeds our ability to effectively manage, curate, analyze, and share it. Data-intensive computing is expected to revolutionize the next-generation software stack. Hadoop, an open source implementation of the MapReduce model provides a way for large data volumes to be seamlessly processed through use of large commodity computers. The inherent parallelization, synchronization and fault-tolerance the model offers, makes it ideal for highly-parallel data-intensive applications. MapReduce and Hadoop have traditionally been used for web data processing and only recently been used for scientific applications. There is a limited understanding on the performance characteristics that scientific data intensive applications can obtain from MapReduce and Hadoop. Thus, it is important to evaluate Hadoop specifically for data-intensive scientific operations -- filter, merge and reorder-- to understand its various design considerations and performance trade-offs. In this paper, we evaluate Hadoop for these data operations in the context of High Performance Computing (HPC) environments to understand the impact of the file system, network and programming modes on performance.
ISBN:9781467328920
1467328928
ISSN:2159-6182
DOI:10.1109/CLOUD.2012.118