An Efficient Filter Strategy for Theta-Join Query in Distributed Environment

Theta-join query is a very popular application in traditional databases, but due to tremendous computation cost and communication cost in distributed environment, it is not efficiently processed for big data. Current researches focus on processing theta-join by using MapReduce framework, which mainl...

Full description

Saved in:
Bibliographic Details
Published in:Proceedings - International Workshops on Parallel Processing pp. 77 - 84
Main Authors: Wenjie Liu, Zhanhuai Li, Yuntao Zhou
Format: Conference Proceeding
Language:English
Published: IEEE 01.08.2017
Subjects:
Big Data
big data query
distributed computing
Distributed databases
Electronic mail
filter strategy
Filtering algorithms
Sparks
theta-join
Transforms
ISSN:1530-2016
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:Theta-join query is a very popular application in traditional databases, but due to tremendous computation cost and communication cost in distributed environment, it is not efficiently processed for big data. Current researches focus on processing theta-join by using MapReduce framework, which mainly consider the overheads of load balance in the network, when the data sets become larger, massive intermediate results lead to high communication cost. In this work, we propose a filter method for theta-join to reduce the computation and communication cost in distributed environment, which can effectively improve the theta-join query. We consider both the load balance in the cluster and the memory cost in the parallel framework. We have implemented our method in a popular general-purpose data processing framework, Spark. The experimental results demonstrate that our method can significantly improve the performance of theta-joins comparing the state-of-art solutions.
ISSN:1530-2016
DOI:10.1109/ICPPW.2017.24