A distributed evolutionary based instance selection algorithm for big data using Apache Spark

Instance selection is an important preprocessing technology in data mining and machine learning. In this paper, we proposed a novel evolutionary based instance selection algorithm for big data. First, we defined a coarse granularity chromosome structure to reduce the size of search space and costs o...

Celý popis

Uložené v:
Podrobná bibliografia
Vydané v:Applied soft computing Ročník 159; s. 111638
Hlavní autori: Qin, Liyang, Wang, Xiaoli, Yin, Linzi, Jiang, Zhaohui
Médium: Journal Article
Jazyk:English
Vydavateľské údaje: Elsevier B.V 01.07.2024
Predmet:
Apache Spark
Big Data
Evolutionary algorithm
Instance selection
Big Data
Instance selection
Evolutionary algorithm
Apache Spark
ISSN:1568-4946, 1872-9681
On-line prístup:Získať plný text
Tagy: Pridať tag
Žiadne tagy, Buďte prvý, kto otaguje tento záznam!
Popis
Shrnutí:Instance selection is an important preprocessing technology in data mining and machine learning. In this paper, we proposed a novel evolutionary based instance selection algorithm for big data. First, we defined a coarse granularity chromosome structure to reduce the size of search space and costs of chromosome operations(recombination and mutation, etc.). Then a stratified evolution strategy was proposed to remove the hyper parameter in classic fitness function and achieve precise control over the reduction ratio of instances. Finally, a sampling-based fitness function was proposed to reduce the time complexity. Experimental results shown that our new algorithm is efficient to complete the instance selection task on data set with millions of instances in minutes-level. The 10-fold cross-validation also proved that the selection results on many datasets have high nearest neighbor classification accuracy. •We proposed a distributed coarse granularity chromosome structure to reduce search space and ensure load balance.•We proposed a stratified evolution strategy to transform multi-objective to multiple single-objective tasks.•We proposed a sampling-based fitness function with less time complexity
ISSN:1568-4946
1872-9681
DOI:10.1016/j.asoc.2024.111638