A distributed evolutionary based instance selection algorithm for big data using Apache Spark

Instance selection is an important preprocessing technology in data mining and machine learning. In this paper, we proposed a novel evolutionary based instance selection algorithm for big data. First, we defined a coarse granularity chromosome structure to reduce the size of search space and costs o...

Full description

Saved in:
Bibliographic Details
Published in:Applied soft computing Vol. 159; p. 111638
Main Authors: Qin, Liyang, Wang, Xiaoli, Yin, Linzi, Jiang, Zhaohui
Format: Journal Article
Language:English
Published: Elsevier B.V 01.07.2024
Subjects:
Apache Spark
Big Data
Evolutionary algorithm
Instance selection
Big Data
Instance selection
Evolutionary algorithm
Apache Spark
ISSN:1568-4946, 1872-9681
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:Instance selection is an important preprocessing technology in data mining and machine learning. In this paper, we proposed a novel evolutionary based instance selection algorithm for big data. First, we defined a coarse granularity chromosome structure to reduce the size of search space and costs of chromosome operations(recombination and mutation, etc.). Then a stratified evolution strategy was proposed to remove the hyper parameter in classic fitness function and achieve precise control over the reduction ratio of instances. Finally, a sampling-based fitness function was proposed to reduce the time complexity. Experimental results shown that our new algorithm is efficient to complete the instance selection task on data set with millions of instances in minutes-level. The 10-fold cross-validation also proved that the selection results on many datasets have high nearest neighbor classification accuracy. •We proposed a distributed coarse granularity chromosome structure to reduce search space and ensure load balance.•We proposed a stratified evolution strategy to transform multi-objective to multiple single-objective tasks.•We proposed a sampling-based fitness function with less time complexity
ISSN:1568-4946
1872-9681
DOI:10.1016/j.asoc.2024.111638