Safe RuleFit: Learning Optimal Sparse Rule Model by Meta Safe Screening

We consider the problem of learning a sparse rule model , a prediction model in the form of a sparse linear combination of rules, where a rule is an indicator function defined over a hyper-rectangle in the input space. Since the number of all possible such rules is extremely large, it has been compu...

Full description

Saved in:
Bibliographic Details
Published in:IEEE Transactions on Pattern Analysis and Machine Intelligence Vol. 45; no. 2; pp. 2330 - 2343
Main Authors: Kato, Hiroki, Hanada, Hiroyuki, Takeuchi, Ichiro
Format: Journal Article
Language:English
Published: United States IEEE 01.02.2023
Institute of Electrical and Electronics Engineers (IEEE)
The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
Subjects:
Analytical models
combinatorial algorithms
convex programming
Dictionaries
knowledge representation formalisms and methods
Learning
Machine learning
Numerical models
Pattern analysis
Prediction models
Predictive models
Random forests
Regression tree analysis
Regularization
Screening
ISSN:0162-8828, 1939-3539, 1939-3539, 2160-9292
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:We consider the problem of learning a sparse rule model , a prediction model in the form of a sparse linear combination of rules, where a rule is an indicator function defined over a hyper-rectangle in the input space. Since the number of all possible such rules is extremely large, it has been computationally intractable to select the optimal set of active rules. In this paper, to solve this difficulty for learning the optimal sparse rule model, we propose Safe RuleFit (SRF) . Our basic idea is to develop meta safe screening (mSS) , which is a non-trivial extension of well-known safe screening (SS) techniques. While SS is used for screening out one feature, mSS can be used for screening out multiple features by exploiting the inclusion-relations of hyper-rectangles in the input space. SRF provides a general framework for fitting sparse rule models for regression and classification, and it can be extended to handle more general sparse regularizations such as group regularization. We demonstrate the advantages of SRF through intensive numerical experiments.
Bibliography:ObjectType-Article-1
SourceType-Scholarly Journals-1
ObjectType-Feature-2
content type line 14
content type line 23
ISSN:0162-8828
1939-3539
1939-3539
2160-9292
DOI:10.1109/TPAMI.2022.3167993