Extension of a Task-Based Model to Functional Programming

Recently, efforts have been made to bring together the areas of high-performance computing (HPC) and massive data processing (Big Data). Traditional HPC frameworks, like COMPSs, are mostly task-based, while popular big-data environments, like Spark, are based on functional programming principles. Th...

Full description

Saved in:
Bibliographic Details
Published in:Proceedings (Symposium on Computer Architecture and High Performance Computing) pp. 64 - 71
Main Authors: Ponce, Lucas M., Lezzi, Daniele, Badia, Rosa M., Guedes, Dorgival
Format: Conference Proceeding
Language:English
Published: IEEE 01.10.2019
Subjects:
Big Data
COMPSs
DataFlow Programming
Libraries
Performance Evaluation
Programming
Runtime
Sparks
Task analysis
ISSN:2643-3001
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:Recently, efforts have been made to bring together the areas of high-performance computing (HPC) and massive data processing (Big Data). Traditional HPC frameworks, like COMPSs, are mostly task-based, while popular big-data environments, like Spark, are based on functional programming principles. The earlier are know for their good performance for regular, matrix-based computations; on the other hand, for fine-grained, data-parallel workloads, the later has often been considered more successful. In this paper we present our experience with the integration of some dataflow techniques into COMPSs, a task-based framework, in an effort to bring together the best aspects of both worlds. We present our API, called DDF, which provides a new data abstraction that addresses the challenges of integrating Big Data application scenarios into COMPSs. DDF has a functional-based interface, similar to many Data Science tools, that allows us to use dynamic evaluation to adapt the task execution in runtime. Besides the performance optimization it provides, the API facilitates the development of applications by experts in the application domain. In this paper we evaluate DDF's effectiveness by comparing the resulting programs to their original versions in COMPSs and Spark. The results show that DDF can improve COMPSs execution time and even outperform Spark in many use cases.
ISSN:2643-3001
DOI:10.1109/SBAC-PAD.2019.00023