Research on Join Operation of Temporal Big Data in Distributed Environment

Distributed system is an ideal choice for processing temporal large data join operation,but the existing distributed system cannot support the original temporal join query and cannot meet the processing requirements of temporal large data with low latency and high throughput.Therefore,a two-level in...

Full description

Saved in:

Bibliographic Details
Published in:	Ji suan ji gong cheng Vol. 45; no. 3; pp. 20 - 25,31
Main Author:	ZHANG Wei,WANG Zhijie
Format:	Journal Article
Language:	Chinese English
Published:	Editorial Office of Computer Engineering 01.03.2019
Subjects:	temporal big data\| distributed memory computing\| temporal join\| two-level index\| partition method\| spark framework
ISSN:	1000-3428
Online Access:	Get full text
Tags:	Add Tag No Tags, Be the first to tag this record!

Description
Summary:	Distributed system is an ideal choice for processing temporal large data join operation,but the existing distributed system cannot support the original temporal join query and cannot meet the processing requirements of temporal large data with low latency and high throughput.Therefore,a two-level index memory solution scheme based on Spark is proposed.The global index is used to prune the distributed partitions,and the local temporal index is used to query the partitions in order to improve the efficiency of data retrieval.A partition method is designed for temporal data to optimize global pruning.Experimental results based on real and synthetic datasets show that the scheme can significantly improve the processing efficiency of temporal join operation.
ISSN:	1000-3428
DOI:	10.19678/j.issn.1000-3428.0052626