Research on Join Operation of Temporal Big Data in Distributed Environment

Distributed system is an ideal choice for processing temporal large data join operation,but the existing distributed system cannot support the original temporal join query and cannot meet the processing requirements of temporal large data with low latency and high throughput.Therefore,a two-level in...

Celý popis

Uložené v:
Podrobná bibliografia
Vydané v:Ji suan ji gong cheng Ročník 45; číslo 3; s. 20 - 25,31
Hlavný autor: ZHANG Wei,WANG Zhijie
Médium: Journal Article
Jazyk:Chinese
English
Vydavateľské údaje: Editorial Office of Computer Engineering 01.03.2019
Predmet:
temporal big data| distributed memory computing| temporal join| two-level index| partition method| spark framework
ISSN:1000-3428
On-line prístup:Získať plný text
Tagy: Pridať tag
Žiadne tagy, Buďte prvý, kto otaguje tento záznam!
Popis
Shrnutí:Distributed system is an ideal choice for processing temporal large data join operation,but the existing distributed system cannot support the original temporal join query and cannot meet the processing requirements of temporal large data with low latency and high throughput.Therefore,a two-level index memory solution scheme based on Spark is proposed.The global index is used to prune the distributed partitions,and the local temporal index is used to query the partitions in order to improve the efficiency of data retrieval.A partition method is designed for temporal data to optimize global pruning.Experimental results based on real and synthetic datasets show that the scheme can significantly improve the processing efficiency of temporal join operation.
ISSN:1000-3428
DOI:10.19678/j.issn.1000-3428.0052626