DDF Library: Enabling functional programming in a task-based model

In recent years, the areas of High-Performance Computing (HPC) and massive data processing (also know as Big Data) have been in a convergence course, since they tend to be deployed on similar hardware. HPC systems have historically performed well in regular, matrix-based computations; on the other h...

Full description

Saved in:
Bibliographic Details
Published in:Journal of parallel and distributed computing Vol. 151; pp. 112 - 124
Main Authors: Ponce, Lucas M., Lezzi, Daniele, Badia, Rosa M., Guedes, Dorgival
Format: Journal Article
Language:English
Published: Elsevier Inc 01.05.2021
Subjects:
Big Data
COMPSs
Data-flow programming
Performance evaluation
Performance evaluation
Big Data
Data-flow programming
COMPSs
ISSN:0743-7315, 1096-0848
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:In recent years, the areas of High-Performance Computing (HPC) and massive data processing (also know as Big Data) have been in a convergence course, since they tend to be deployed on similar hardware. HPC systems have historically performed well in regular, matrix-based computations; on the other hand, Big Data problems have often excelled in fine-grained, data parallel workloads. While HPC programming is mostly task-based, like COMPSs, popular Big Data environments, like Spark, adopt the functional programming paradigm. A careful analysis shows that there are pros and cons to both approaches, and integrating them may yield interesting results. With that reasoning in mind, we have developed DDF, an API and library for COMPSs that allows developers to use Big Data techniques while using that HPC environment. DDF has a functional-based interface, similar to many Data Science tools, that allows us to use dynamic evaluation to adapt the task execution in run time. It brings some of the qualities of Big Data programming, making it easier for application domain experts to write Data Analysis jobs. In this article we discuss the API and evaluate the impact of the techniques used in its implementation that allow a more efficient COMPSs execution. In addition, we present a performance comparison with Spark in several application patterns. The results show that each technique significantly impacts the performance, allowing COMPSs to outperform Spark in many use cases. •Big Data techniques are evaluated in a HPC system.•An extensible API is proposed to allow high-level functional programming in COMPSs.•A performance comparison between COMPSs and Spark is made for different applications patterns.
ISSN:0743-7315
1096-0848
DOI:10.1016/j.jpdc.2021.02.009