On the Dual Formulation of Boosting Algorithms

We study boosting algorithms from a new perspective. We show that the Lagrange dual problems of ℓ 1 -norm-regularized AdaBoost, LogitBoost, and soft-margin LPBoost with generalized hinge loss are all entropy maximization problems. By looking at the dual problems of these boosting algorithms, we show...

Full description

Saved in:
Bibliographic Details
Published in:IEEE transactions on pattern analysis and machine intelligence Vol. 32; no. 12; pp. 2216 - 2231
Main Authors: Shen, Chunhua, Li, Hanxi
Format: Journal Article
Language:English
Published: Los Alamitos, CA IEEE 01.12.2010
IEEE Computer Society
The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
Subjects:
AdaBoost
Algorithms
Applied sciences
Artificial intelligence
Australia
Boosting
Classification
Computer science; control theory; systems
Construction
Convergence
Data processing. List processing. Character string processing
Entropy
entropy maximization
Exact sciences and technology
Fasteners
Formulations
Lagrange duality
Lagrangian functions
Linear programming
LogitBoost
LPBoost
Machine learning algorithms
Maximization
Memory organisation. Data processing
Optimization
Software
Support vector machine classification
Support vector machines
Data analysis
Linear programming
LPBoost
Optimization
Variance
Aggregate model
Lagrange duality
AdaBoost
Supervised learning
Classification
Convergence rate
LogitBoost
entropy maximization
Numerical convergence
ISSN:0162-8828, 1939-3539, 1939-3539
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:We study boosting algorithms from a new perspective. We show that the Lagrange dual problems of ℓ 1 -norm-regularized AdaBoost, LogitBoost, and soft-margin LPBoost with generalized hinge loss are all entropy maximization problems. By looking at the dual problems of these boosting algorithms, we show that the success of boosting algorithms can be understood in terms of maintaining a better margin distribution by maximizing margins and at the same time controlling the margin variance. We also theoretically prove that approximately, ℓ 1 -norm-regularized AdaBoost maximizes the average margin, instead of the minimum margin. The duality formulation also enables us to develop column-generation-based optimization algorithms, which are totally corrective. We show that they exhibit almost identical classification results to that of standard stagewise additive boosting algorithms but with much faster convergence rates. Therefore, fewer weak classifiers are needed to build the ensemble using our proposed optimization technique.
Bibliography:ObjectType-Article-1
SourceType-Scholarly Journals-1
ObjectType-Feature-2
content type line 14
content type line 23
ObjectType-Article-2
ObjectType-Feature-1
ISSN:0162-8828
1939-3539
1939-3539
DOI:10.1109/TPAMI.2010.47