A new feature selection algorithm based on binomial hypothesis testing for spam filtering

Content-based spam filtering is a binary text categorization problem. To improve the performance of the spam filtering, feature selection, as an important and indispensable means of text categorization, also plays an important role in spam filtering. We proposed a new method, named Bi-Test, which ut...

Full description

Saved in:
Bibliographic Details
Published in:Knowledge-Based Systems Vol. 24; no. 6; pp. 904 - 914
Main Authors: Yang, Jieming, Liu, Yuanning, Liu, Zhen, Zhu, Xiaodong, Zhang, Xiaoxu
Format: Journal Article
Language:English
Japanese
Published: Elsevier B.V 01.08.2011
Elsevier BV
Subjects:
Algorithms
Binomial distribution
Binomial hypothesis testing
Email
Feature selection
Spam filtering
Spamming
Text categorization
Binomial distribution
Feature selection
Binomial hypothesis testing
Spam filtering
Text categorization
ISSN:0950-7051, 1872-7409
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:Content-based spam filtering is a binary text categorization problem. To improve the performance of the spam filtering, feature selection, as an important and indispensable means of text categorization, also plays an important role in spam filtering. We proposed a new method, named Bi-Test, which utilizes binomial hypothesis testing to estimate whether the probability of a feature belonging to the spam satisfies a given threshold or not. We have evaluated Bi-Test on six benchmark spam corpora (pu1, pu2, pu3, pua, lingspam and CSDMC2010), using two classification algorithms, Naïve Bayes (NB) and Support Vector Machines (SVM), and compared it with four famous feature selection algorithms (information gain, χ 2-statistic, improved Gini index and Poisson distribution). The experiments show that Bi-Test performs significantly better than χ 2-statistic and Poisson distribution, and produces comparable performance with information gain and improved Gini index in terms of F1 measure when Naïve Bayes classifier is used; it achieves comparable performance with the other methods when SVM classifier is used. Moreover, Bi-Test executes faster than the other four algorithms.
Bibliography:ObjectType-Article-1
SourceType-Scholarly Journals-1
ObjectType-Feature-2
content type line 23
ISSN:0950-7051
1872-7409
DOI:10.1016/j.knosys.2011.04.006