Parallelizing user–defined functions in the ETL workflow using orchestration style sheets

Today’s ETL tools provide capabilities to develop custom code as user-defined functions (UDFs) to extend the expressiveness of the standard ETL operators. However, while this allows us to easily add new functionalities, it also comes with the risk that the custom code is not intended to be optimized...

Celý popis

Uloženo v:
Podrobná bibliografie
Vydáno v:International journal of applied mathematics and computer science Ročník 29; číslo 1; s. 69 - 79
Hlavní autoři: Ali, Syed Muhammad Fawad, Mey, Johannes, Thiele, Maik
Médium: Journal Article
Jazyk:angličtina
Vydáno: Zielona Góra Sciendo 01.03.2019
De Gruyter Brill Sp. z o.o., Paradigm Publishing Services
Témata:
Configuration management
Configurations
ETL workflow
parallel algorithmic skeletons
parallel ETL operators
Parallel processing
Task complexity
user-defined functions
Workflow
ISSN:2083-8492, 1641-876X, 2083-8492
On-line přístup:Získat plný text
Tagy: Přidat tag
Žádné tagy, Buďte první, kdo vytvoří štítek k tomuto záznamu!
Popis
Shrnutí:Today’s ETL tools provide capabilities to develop custom code as user-defined functions (UDFs) to extend the expressiveness of the standard ETL operators. However, while this allows us to easily add new functionalities, it also comes with the risk that the custom code is not intended to be optimized, e.g., by parallelism, and for this reason, it performs poorly for data-intensive ETL workflows. In this paper we present a novel framework, which allows the ETL developer to choose a design pattern in order to write parallelizable code and generates a configuration for the UDFs to be executed in a distributed environment. This enables ETL developers with minimum expertise in distributed and parallel computing to develop UDFs without taking care of parallelization configurations and complexities. We perform experiments on large-scale datasets based on TPC-DS and BigBench. The results show that our approach significantly reduces the effort of ETL developers and at the same time generates efficient parallel configurations to support complex and data-intensive ETL tasks.
Bibliografie:ObjectType-Article-1
SourceType-Scholarly Journals-1
ObjectType-Feature-2
content type line 14
ISSN:2083-8492
1641-876X
2083-8492
DOI:10.2478/amcs-2019-0005