Extension of a Task-Based Model to Functional Programming

Recently, efforts have been made to bring together the areas of high-performance computing (HPC) and massive data processing (Big Data). Traditional HPC frameworks, like COMPSs, are mostly task-based, while popular big-data environments, like Spark, are based on functional programming principles. Th...

Celý popis

Uloženo v:
Podrobná bibliografie
Vydáno v:Proceedings (Symposium on Computer Architecture and High Performance Computing) s. 64 - 71
Hlavní autoři: Ponce, Lucas M., Lezzi, Daniele, Badia, Rosa M., Guedes, Dorgival
Médium: Konferenční příspěvek
Jazyk:angličtina
Vydáno: IEEE 01.10.2019
Témata:
Big Data
COMPSs
DataFlow Programming
Libraries
Performance Evaluation
Programming
Runtime
Sparks
Task analysis
ISSN:2643-3001
On-line přístup:Získat plný text
Tagy: Přidat tag
Žádné tagy, Buďte první, kdo vytvoří štítek k tomuto záznamu!
Popis
Shrnutí:Recently, efforts have been made to bring together the areas of high-performance computing (HPC) and massive data processing (Big Data). Traditional HPC frameworks, like COMPSs, are mostly task-based, while popular big-data environments, like Spark, are based on functional programming principles. The earlier are know for their good performance for regular, matrix-based computations; on the other hand, for fine-grained, data-parallel workloads, the later has often been considered more successful. In this paper we present our experience with the integration of some dataflow techniques into COMPSs, a task-based framework, in an effort to bring together the best aspects of both worlds. We present our API, called DDF, which provides a new data abstraction that addresses the challenges of integrating Big Data application scenarios into COMPSs. DDF has a functional-based interface, similar to many Data Science tools, that allows us to use dynamic evaluation to adapt the task execution in runtime. Besides the performance optimization it provides, the API facilitates the development of applications by experts in the application domain. In this paper we evaluate DDF's effectiveness by comparing the resulting programs to their original versions in COMPSs and Spark. The results show that DDF can improve COMPSs execution time and even outperform Spark in many use cases.
ISSN:2643-3001
DOI:10.1109/SBAC-PAD.2019.00023