How to Continually Adapt Text-to-Image Diffusion Models for Flexible Customization?
Dong, Jiahua Liang, Wenqi Li, Hongliu Zhang, Duzhen Cao, Meng Ding, Henghui Khan, Salman Khan, Fahad Shahbaz
Dong, Jiahua
Liang, Wenqi
Li, Hongliu
Zhang, Duzhen
Cao, Meng
Ding, Henghui
Khan, Salman
Khan, Fahad Shahbaz
Supervisor
Department
Computer Vision
Embargo End Date
Type
Conference proceeding
Date
2024
License
Language
English
Collections
Research Projects
Organizational Units
Journal Issue
Abstract
Custom diffusion models (CDMs) have attracted widespread attention due to their astonishing generative ability for personalized concepts. However, most existing CDMs unreasonably assume that personalized concepts are fixed and cannot change over time. Moreover, they heavily suffer from catastrophic forgetting and concept neglect on old personalized concepts when continually learning a series of new concepts. To address these challenges, we propose a novel Concept-Incremental text-to-image Diffusion Model (CIDM), which can resolve catastrophic forgetting and concept neglect to learn new customization tasks in a concept-incremental manner. Specifically, to surmount the catastrophic forgetting of old concepts, we develop a concept consolidation loss and an elastic weight aggregation module. They can explore task-specific and task-shared knowledge during training, and aggregate all low-rank weights of old concepts based on their contributions during inference. Moreover, in order to address concept neglect, we devise a context-controllable synthesis strategy that leverages expressive region features and noise estimation to control the contexts of generated images according to user conditions. Experiments validate that our CIDM surpasses existing custom diffusion models. The source codes are available at https://github.com/JiahuaDong/CIFC.
Citation
J. Dong et al., “How to Continually Adapt Text-to-Image Diffusion Models for Flexible Customization?,” Adv Neural Inf Process Syst, vol. 37, pp. 130057–130083, Dec. 2024, Accessed: Mar. 24, 2025. [Online]. Available: https://github.com/JiahuaDong/CIFC.
Source
Advances in Neural Information Processing Systems (NeurIPS 2024)
Conference
Keywords
Continual adaptation, Text-to-image diffusion models, Custom diffusion models (CDMs), Catastrophic forgetting, Concept-incremental learning
Subjects
Source
Publisher
NEURIPS