Loading...
Learning discrete concepts in latent hierarchical models
Kong, Lingjing Chen, Guangyi Huang, Biwei Xing, Eric Chi, Yuejie Zhang, Kun
Kong, Lingjing
Chen, Guangyi
Huang, Biwei
Xing, Eric
Chi, Yuejie
Zhang, Kun
Supervisor
Department
Machine Learning
Embargo End Date
Type
Conference proceeding
Date
2024
License
Language
English
Collections
Research Projects
Organizational Units
Journal Issue
Abstract
Learning concepts from natural high-dimensional data (e.g., images) holds potential in building human-aligned and interpretable machine learning models. Despite its encouraging prospect, formalization and theoretical insights into this crucial task are still lacking. In this work, we formalize concepts as discrete latent causal variables that are related via a hierarchical causal model that encodes different abstraction levels of concepts embedded in high-dimensional data (e.g., a dog breed and its eye shapes in natural images). We formulate conditions to facilitate the identification of the proposed causal model, which reveals when learning such concepts from unsupervised data is possible. Our conditions permit complex causal hierarchical structures beyond latent trees and multi-level directed acyclic graphs in prior work and can handle high-dimensional, continuous observed variables, which is well-suited for unstructured data modalities such as images. We substantiate our theoretical claims with synthetic data experiments. Further, we discuss our theory's implications for understanding the underlying mechanisms of latent diffusion models and provide corresponding empirical evidence for our theoretical insights.
Citation
L. Kong, G. Chen, B. Huang, E. P. Xing, Y. Chi, and K. Zhang, “Learning Discrete Concepts in Latent Hierarchical Models,” Adv Neural Inf Process Syst, vol. 37, pp. 36938–36975, Dec. 2024.
Source
Advances in Neural Information Processing Systems (NeurIPS 2024)
Conference
Keywords
Learning discrete concepts, Latent hierarchical models, Causal variables, High-dimensional data, Concept identification
Subjects
Source
Publisher
NEURIPS