Ensemble Linear Interpolators: The Role of Ensembling
Wu, Mingqi Sun, Qiang
Wu, Mingqi
Sun, Qiang
Author
Supervisor
Department
Statistics and Data Science
Embargo End Date
Type
Journal article
Date
2025
License
Language
English
Collections
Research Projects
Organizational Units
Journal Issue
Abstract
Interpolators are unstable. For example, the minimum
norm least squares interpolator exhibits unbounded test errors when dealing with noisy data. In this paper, we study how ensemble stabilizes and thus improves the generalization performance, measured by the out-of-sample prediction risk, of an individual interpolator. We focus on bagged linear interpolators, as bagging is a popular randomization-based ensemble method that can be implemented in parallel. We introduce the multiplier-bootstrap-based bagged least squares estimator, which can then be formulated as an average of the sketched least squares estimators. The proposed multiplier bootstrap encompasses the classical bootstrap with replacement as a special case, along with a more intriguing variant which we call the Bernoulli bootstrap. Focusing on the proportional regime where the sample size scales proportionally with the feature dimensionality, we investigate the out-of-sample prediction risks of the sketched and bagged least squares estimators in both underparametrized and overparameterized regimes. Our results reveal the statistical roles of sketching and bagging. In particular, sketching modifies the aspect ratio and shifts the interpolation threshold of the minimum
norm estimator. However, the risk of the sketched estimator continues to be unbounded around the interpolation threshold due to excessive variance. In stark contrast, bagging effectively mitigates this variance, leading to a bounded limiting out-of-sample prediction risk. To further understand this stability improvement property, we establish that bagging acts as a form of implicit regularization, substantiated by the equivalence of the bagged estimator with its explicitly regularized counterpart. We also discuss several extensions.
Citation
M. Wu and Q. Sun, “Ensemble Linear Interpolators: The Role of Ensembling,” https://doi.org/10.1137/24M1642548, vol. 7, no. 2, pp. 438–467, Apr. 2025, doi: 10.1137/24M1642548.
Source
SIAM Journal on Mathematics of Data Science
Conference
Keywords
Bagging, Ensemble, Interpolators, Random matrix theory, Sketching
Subjects
Source
Publisher
SIAM PUBLICATIONS