A Hybrid Stochastic-Deterministic Minibatch Proximal Gradient Method for Efficient Optimization and Generalization - Zhou, Pan; Yuan, Xiao-Tong; Lin, Zhouchen; Hoi, Steven C.H. (IEEE transactions on pattern analysis and machine intelligence, 2022)
Despite the success of stochastic variance-reduced gradient (SVRG) algorithms in solving large-scale problems, their stochastic gradient complexity often scales linearly with data size and is expensive for huge data. Accordingly, we propose a hybrid stochastic-deterministic minibatch proximal gradient ( HSDMPG ) algorithm for strongly convex problems with linear prediction structure, e.g., least squares and logistic/softmax regression. HSDMPG enjoys improved computational complexity that is data-size-independent for large-scale problems. It iteratively samples an evolving minibatch of individual losses to estimate the original problem, and can efficiently minimize the sampled subproblems. For strongly convex loss of n n<inline-graphic xlink:href="zhou-ieq1-3087328.gif"/> components, HSDMPG attains an \epsilon ε<inline-graphic xlink:href="zhou-ieq2-3087328.gif"/> -optimization-error within \mathcal {O} \left(\kappa \log ^{\zeta +1}\left(\frac{1}{\epsilon }\right)\frac{1}{\epsilon }\bigwedge n\log ^{\zeta }\left(\frac{1}{\epsilon }\right)\right) Oκlogζ+11ε1ε⋀nlogζ1ε<inline-graphic xlink:href="zhou-ieq3-3087328.gif"/> stochastic gradient evaluations, where \kappa κ<inline-graphic xlink:href="zhou-ieq4-3087328.gif"/> is condition number, \zeta =1 ζ=1<inline-graphic xlink:href="zhou-ieq5-3087328.gif"/> for quadratic loss and \zeta =2 ζ=2<inline-graphic xlink:href="zhou-ieq6-3087328.gif"/> for generic loss. For large-scale problems, our complexity outperforms those of SVRG-type algorithms with/without dependence on data size. Particularly, when \epsilon =\mathcal {O}(1/\sqrt{n}) ε=O(1/n)<inline-graphic xlink:href="zhou-ieq7-3087328.gif"/> which matches the intrinsic excess error of a learning model and is sufficient for generalization, our complexity for quadratic and generic losses is respectively \mathcal {O} (n^{0.5}\log ^{2}(n)) O(n0.5log2(n))<inline-graphic xlink:href="zhou-ieq8-3087328.gif"/> and \mathcal {O} (n^{0.5}\log ^{3}(n)) O(n0.5log3(n))<inline-graphic xlink:href="zhou-ieq9-3087328.gif"/> , which for the first time achieves optimal generalization in less than a single pass over data. Besides, we extend HSDMPG to online strongly convex problems and prove its higher efficiency over the prior algorithms. Numerical results demonstrate the computational advantages of HSDMPG .