Stagewise Training With Exponentially Growing Training Sets
Gu, Bin Alquabeh, Hilal De Vazelhes, William Huo, Zhouyuan Huang, Heng
Gu, Bin
Alquabeh, Hilal
De Vazelhes, William
Huo, Zhouyuan
Huang, Heng
Supervisor
Department
Machine Learning
Embargo End Date
Type
Journal article
Date
2025
License
Language
English
Collections
Research Projects
Organizational Units
Journal Issue
Abstract
In the world of big data, training large-scale machine learning problems has gained considerable attention. Numerous innovative optimization strategies have been presented in recent years to accelerate the large-scale training process. However, the possibility of further accelerating the training process of various optimization algorithms remains an unresolved subject. To begin addressing this difficult problem, we exploit the researched findings that when training data are independent and identically distributed, the learning problem on a smaller dataset is not significantly different from the original one. Upon that, we propose a stagewise training technique that grows the size of the training set exponentially while solving nonsmooth subproblem. We demonstrate that our stagewise training via exponentially growing the size of the training sets (STEGSs) are compatible with a large number of proximal gradient descent and gradient hard thresholding (GHT) techniques. Interestingly, we demonstrate that STEGS can greatly reduce overall complexity while maintaining statistical accuracy or even surpassing the intrinsic error introduced by GHT approaches. In addition, we analyze the effect of the training data growth rate on the overall complexity. The practical results of applying l2,1 - and l0 -norms to a variety of large-scale real-world datasets not only corroborate our theories but also demonstrate the benefits of our STEGS framework. © 2012 IEEE.
Citation
B. Gu, H. AlQuabeh, W. de Vazelhes, Z. Huo, and H. Huang, “Stagewise Training With Exponentially Growing Training Sets,” IEEE Trans Neural Netw Learn Syst, vol. 36, no. 4, pp. 6148–6158, 2025, doi: 10.1109/TNNLS.2024.3402108
Source
IEEE Transactions on Neural Networks and Learning Systems
Conference
Keywords
Gradient hard thresholding (GHT) algorithms, proximal algorithms, stagewise training strategy, stochastic optimization
Subjects
Source
Publisher
IEEE