Design and implementation of reconfigurable acceleration for in-memory distributed big data computing

Apache Spark is an efficient distributed computing framework for big data processing. It supports in-memory computation of RDDs (Resilient Distributed Datasets) and provides a provision of reusability, fault tolerance, and real-time stream processing. However, the tasks in Spark framework are only p...

Full description

Saved in:
Bibliographic Details
Published in:Future generation computer systems Vol. 92; pp. 68 - 75
Main Authors: Hou, Junjie, Zhu, Yongxin, Du, Sen, Song, Shijin
Format: Journal Article
Language:English
Published: Elsevier B.V 01.03.2019
Subjects:
Distributed computing
Energy efficiency
FPGA
High performance
OpenCL
Python
Spark
High performance
FPGA
Energy efficiency
Spark
OpenCL
Distributed computing
Python
ISSN:0167-739X, 1872-7115
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:Apache Spark is an efficient distributed computing framework for big data processing. It supports in-memory computation of RDDs (Resilient Distributed Datasets) and provides a provision of reusability, fault tolerance, and real-time stream processing. However, the tasks in Spark framework are only performed on CPU. The low degree of parallelism and power inefficiency of CPU may restrict the performance and scalability of the cluster. In order to improve the performance and power dissipation of the data center, heterogeneous accelerators such as FPGA, GPU, MIC (Many Integrated Core) exhibit more efficient performance than general-purpose processors in big data processing. In this work, we propose a framework to integrate FPGA accelerators into a Spark cluster, which achieves performance improvement and power dissipation reduction for distributed applications. We propose a method for connecting Spark with OpenCL application which is a standard for cross-platform, parallel programming of diverse processors and widely used in heterogeneous computing, and use FPGA to accelerate the Spark tasks developed with Python. We illustrate the performance and the energy efficiency of FPGA based Spark framework with a case study of K-means algorithm acceleration. The results show that FPGA based Spark implementation achieves 3.5x speedup and 4.06x energy efficiency over original Spark framework. •The methodology of integrating FPGA accelerator to the Spark framework is proposed.•Optimizing the execution of K-means clustering in FPGA based Spark implementation.•A case study of K-means algorithm acceleration on FPGA based Spark framework.
ISSN:0167-739X
1872-7115
DOI:10.1016/j.future.2018.09.049