LotusSQL: SQL engine for high-performance big data systems

In recent years, Apache Spark has become the de facto standard for big data processing. SparkSQL is a module offering support for relational analysis on Spark with Structured Query Language (SQL). SparkSQL provides convenient data processing interfaces. Despite its efficient optimizer, SparkSQL stil...

Celý popis

Uloženo v:
Podrobná bibliografie
Vydáno v:Big Data Mining and Analytics Ročník 4; číslo 4; s. 252 - 265
Hlavní autoři: Li, Xiaohan, Yu, Bowen, Feng, Guanyu, Wang, Haojie, Chen, Wenguang
Médium: Journal Article
Jazyk:angličtina
Vydáno: Beijing Tsinghua University Press 01.12.2021
Témata:
Big Data
c++
C++ (programming language)
Data processing
Data systems
Datasets
Native languages
Queries
Query languages
query optimization
Structured Query Language-SQL
Virtual environments
ISSN:2096-0654, 2097-406X
On-line přístup:Získat plný text
Tagy: Přidat tag
Žádné tagy, Buďte první, kdo vytvoří štítek k tomuto záznamu!
Popis
Shrnutí:In recent years, Apache Spark has become the de facto standard for big data processing. SparkSQL is a module offering support for relational analysis on Spark with Structured Query Language (SQL). SparkSQL provides convenient data processing interfaces. Despite its efficient optimizer, SparkSQL still suffers from the inefficiency of Spark resulting from Java virtual machine and the unnecessary data serialization and deserialization. Adopting native languages such as C++ could help to avoid such bottlenecks. Benefiting from a bare-metal runtime environment and template usage, systems with C++ interfaces usually achieve superior performance. However, the complexity of native languages also increases the required programming and debugging efforts. In this work, we present LotusSQL, an engine to provide SQL support for dataset abstraction on a native backend Lotus. We employ a convenient SQL processing framework to deal with frontend jobs. Advanced query optimization technologies are added to improve the quality of execution plans. Above the storage design and user interface of the compute engine, LotusSQL implements a set of structured dataset operations with high efficiency and integrates them with the frontend. Evaluation results show that LotusSQL achieves a speedup of up to9×in certain queries and outperforms Spark SQL in a standard query benchmark by more than2×on average.
Bibliografie:ObjectType-Article-1
SourceType-Scholarly Journals-1
ObjectType-Feature-2
content type line 14
ISSN:2096-0654
2097-406X
DOI:10.26599/BDMA.2021.9020009