An Efficient Filter Strategy for Theta-Join Query in Distributed Environment

Theta-join query is a very popular application in traditional databases, but due to tremendous computation cost and communication cost in distributed environment, it is not efficiently processed for big data. Current researches focus on processing theta-join by using MapReduce framework, which mainl...

Ausführliche Beschreibung

Gespeichert in:
Bibliographische Detailangaben
Veröffentlicht in:Proceedings - International Workshops on Parallel Processing S. 77 - 84
Hauptverfasser: Wenjie Liu, Zhanhuai Li, Yuntao Zhou
Format: Tagungsbericht
Sprache:Englisch
Veröffentlicht: IEEE 01.08.2017
Schlagworte:
Big Data
big data query
distributed computing
Distributed databases
Electronic mail
filter strategy
Filtering algorithms
Sparks
theta-join
Transforms
ISSN:1530-2016
Online-Zugang:Volltext
Tags: Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
Beschreibung
Zusammenfassung:Theta-join query is a very popular application in traditional databases, but due to tremendous computation cost and communication cost in distributed environment, it is not efficiently processed for big data. Current researches focus on processing theta-join by using MapReduce framework, which mainly consider the overheads of load balance in the network, when the data sets become larger, massive intermediate results lead to high communication cost. In this work, we propose a filter method for theta-join to reduce the computation and communication cost in distributed environment, which can effectively improve the theta-join query. We consider both the load balance in the cluster and the memory cost in the parallel framework. We have implemented our method in a popular general-purpose data processing framework, Spark. The experimental results demonstrate that our method can significantly improve the performance of theta-joins comparing the state-of-art solutions.
ISSN:1530-2016
DOI:10.1109/ICPPW.2017.24