Sparse learning via Boolean relaxations

We introduce novel relaxations for cardinality-constrained learning problems, including least-squares regression as a special but important case. Our approach is based on reformulating a cardinality-constrained problem exactly as a Boolean program, to which standard convex relaxations such as the La...

Full description

Saved in:
Bibliographic Details
Published in:Mathematical programming Vol. 151; no. 1; pp. 63 - 87
Main Authors: Pilanci, Mert, Wainwright, Martin J., El Ghaoui, Laurent
Format: Journal Article
Language:English
Published: Berlin/Heidelberg Springer Berlin Heidelberg 01.06.2015
Springer Nature B.V
Subjects:
Artificial intelligence
Boolean
Calculus of Variations and Optimal Control; Optimization
Combinatorics
Control theory
Electrical engineering
Full Length Paper
Machine learning
Mathematical and Computational Physics
Mathematical Methods in Physics
Mathematics
Mathematics and Statistics
Mathematics of Computing
Numerical Analysis
Optimization
Sparsity
Studies
Support vector machines
Theoretical
United States > US
68T05 Learning and adaptive systems
90C25 Convex programming
90C09 Boolean programming
Sparsity
90C06 Large-scale problems
Machine learning
Convex relaxation
Combinatorial optimization
Regularization
ISSN:0025-5610, 1436-4646
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:We introduce novel relaxations for cardinality-constrained learning problems, including least-squares regression as a special but important case. Our approach is based on reformulating a cardinality-constrained problem exactly as a Boolean program, to which standard convex relaxations such as the Lasserre and Sherali-Adams hierarchies can be applied. We analyze the first-order relaxation in detail, deriving necessary and sufficient conditions for exactness in a unified manner. In the special case of least-squares regression, we show that these conditions are satisfied with high probability for random ensembles satisfying suitable incoherence conditions, similar to results on ℓ 1 -relaxations. In contrast to known methods, our relaxations yield lower bounds on the objective, and it can be verified whether or not the relaxation is exact. If it is not, we show that randomization based on the relaxed solution offers a principled way to generate provably good feasible solutions. This property enables us to obtain high quality estimates even if incoherence conditions are not met, as might be expected in real datasets. We numerically illustrate the performance of the relaxation-randomization strategy in both synthetic and real high-dimensional datasets, revealing substantial improvements relative to ℓ 1 -based methods and greedy selection heuristics.
Bibliography:SourceType-Scholarly Journals-1
ObjectType-Feature-1
content type line 14
ISSN:0025-5610
1436-4646
DOI:10.1007/s10107-015-0894-1