Towards Understanding Extrapolation: a Causal Lens
Kong, Lingjing Chen, Guangyi Stojanov, Petar Li, Haoxuan Xing, Eric P. Zhang, Kun
Kong, Lingjing
Chen, Guangyi
Stojanov, Petar
Li, Haoxuan
Xing, Eric P.
Zhang, Kun
Supervisor
Department
Machine Learning
Embargo End Date
Type
Conference proceeding
Date
2024
License
Language
English
Collections
Research Projects
Organizational Units
Journal Issue
Abstract
Canonical work handling distribution shifts typically necessitates an entire target distribution that lands inside the training distribution.However, practical scenarios often involve only a handful target samples, potentially lying outside the training support, which requires the capability of extrapolation.In this work, we aim to provide a theoretical understanding of when extrapolation is possible and offer principled methods to achieve it without requiring an on-support target distribution.To this end, we formulate the extrapolation problem with a latent-variable model that embodies the minimal change principle in causal mechanisms.Under this formulation, we cast the extrapolation problem into a latent-variable identification problem.We provide realistic conditions on shift properties and the estimation objectives that lead to identification even when only one off-support target sample is available, tackling the most challenging scenarios.Our theory reveals the intricate interplay between the underlying manifold's smoothness and the shift properties.We showcase how our theoretical results inform the design of practical adaptation algorithms. Through experiments on both synthetic and real-world data, we validate our theoretical findings and their practical implications.
Citation
L. Kong, G. Chen, P. Stojanov, H. Li, E. P. Xing, and K. Zhang, “Towards Understanding Extrapolation: a Causal Lens,” Adv Neural Inf Process Syst, vol. 37, pp. 123534–123562, Dec. 2024.
Source
Advances in Neural Information Processing Systems (NeurIPS 2024)
Conference
Keywords
Extrapolation, Distribution shifts, Latent-variable models, Minimal change principle, Test-time adaptation
Subjects
Source
Publisher
NEURIPS