Classification via Minimum Incremental Coding Length

We present a simple new criterion for classification, based on principles from lossy data compression. The criterion assigns a test sample to the class that uses the minimum number of additional bits to code the test sample, subject to an allowable distortion. We demonstrate the asymptotic optimalit...

Full description

Saved in:
Bibliographic Details
Published in:SIAM journal on imaging sciences Vol. 2; no. 2; pp. 367 - 395
Main Authors: Wright, John, Ma, Yi, Tao, Yangyu, Lin, Zhouchen, Shum, Heung-Yeung
Format: Journal Article
Language:English
Published: Philadelphia Society for Industrial and Applied Mathematics 01.01.2009
Subjects:
Asymptotic properties
Classification
Classifiers
Coding
Criteria
Matlab
Support vector machines
ISSN:1936-4954, 1936-4954
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:We present a simple new criterion for classification, based on principles from lossy data compression. The criterion assigns a test sample to the class that uses the minimum number of additional bits to code the test sample, subject to an allowable distortion. We demonstrate the asymptotic optimality of this criterion for Gaussian distributions and analyze its relationships to classical classifiers. The theoretical results clarify the connections between our approach and popular classifiers such as maximum a posteriori (MAP), regularized discriminant analysis (RDA), $k$-nearest neighbor ($k$-NN), and support vector machine (SVM), as well as unsupervised methods based on lossy coding. Our formulation induces several good effects on the resulting classifier. First, minimizing the lossy coding length induces a regularization effect which stabilizes the (implicit) density estimate in a small sample setting. Second, compression provides a uniform means of handling classes of varying dimension. The new criterion and its kernel and local versions perform competitively on synthetic examples, as well as on real imagery data such as handwritten digits and face images. On these problems, the performance of our simple classifier approaches the best reported results, without using domain-specific information. All MATLAB code and classification results are publicly available for peer evaluation at http://perception.csl.uiuc.edu/coding/home.htm.
Bibliography:ObjectType-Article-1
SourceType-Scholarly Journals-1
content type line 14
ObjectType-Article-2
ObjectType-Feature-1
content type line 23
ISSN:1936-4954
1936-4954
DOI:10.1137/070707312