Predictive Subgroup Logistic Regression for Classification with Unobserved Heterogeneity - Chen, Kun; Huang, Rui; Tong, Zhiwei (Statistics and computing, 2025)
Unobserved heterogeneity refers to the variation among subjects that is not accounted for by the observed features used in a model. Its presence poses a substantial challenge to statistical modeling. This study introduces the Predictive Subgroup Logistic Regression (PSLR) model, which extends the conventional logistic regression and is specifically designed to address unobserved heterogeneity in classification problems. The PSLR model incorporates subject-specific intercepts in the log odds, fitted through a penalized likelihood approach with a concave pairwise fusion penalty. A novel two-step procedure is developed to facilitate the out-of-sample predictions for new subjects whose subgroup membership labels are unknown. This procedure allows the PSLR model to perform both inferential and predictive tasks. Through extensive simulation studies and an empirical application to a customer churn dataset in the telecommunications industry, the PSLR model not only demonstrates great performance in various aggregate accuracy metrics but also achieves a balanced effectiveness in sensitivity and specificity.