Optimization of frequent item set mining parallelization algorithm based on spark platform

In this paper, we propose a new method that combines the parallelism of the Spark-based platform with fast frequent mining, called STB_Apriori. Previous research has shown that traditional frequent itemset mining algorithms have high overhead when faced with large datasets and high-dimensional data...

Celý popis

Uložené v:
Podrobná bibliografia
Vydané v:Information retrieval (Boston) Ročník 27; číslo 1; s. 38
Médium: Journal Article
Jazyk:English
Vydavateľské údaje: Dordrecht Springer Nature B.V 02.11.2024
Predmet:
Algorithms
Boolean
Clustering
Data compression
Data mining
Datasets
Distributed processing
Efficiency
Parallel processing
Python
Social networks
User requirements
ISSN:1386-4564, 1573-7659
On-line prístup:Získať plný text
Tagy: Pridať tag
Žiadne tagy, Buďte prvý, kto otaguje tento záznam!
Popis
Shrnutí:In this paper, we propose a new method that combines the parallelism of the Spark-based platform with fast frequent mining, called STB_Apriori. Previous research has shown that traditional frequent itemset mining algorithms have high overhead when faced with large datasets and high-dimensional data computation, and generate a large number of candidate itemsets; at the same time, when faced with diverse user requirements, they often generate very sparse and diverse data. In order to solve the problem of fast mining of massive data, our idea originates from the capability of Spark distributed computing and the common optimisation ideas in Apriori mining, by using the efficient operator BitSet to achieve transaction compression, bit storage and data manipulation by Boolean matrices, and at the same time by parallelising the processing and optimising the algorithmic logic to achieve fast and frequent mining. In experiments on real-world datasets, our model consistently outperforms five widely used methods by a significant margin on very large data and maintains its excellence in the remaining cases, proving its effectiveness on real-world tasks, while further analysis shows that increasing the number of distributed nodes also incrementally and continuously improves performance.
Bibliografia:ObjectType-Article-1
SourceType-Scholarly Journals-1
ObjectType-Feature-2
content type line 14
ISSN:1386-4564
1573-7659
DOI:10.1007/s10791-024-09470-5