Layered Rendering Diffusion Model for Controllable Zero-Shot Image Synthesis
Qi, Zipeng Huang, Guoxi Liu, Chenyang Ye, Fei
Qi, Zipeng
Huang, Guoxi
Liu, Chenyang
Ye, Fei
Supervisor
Department
Machine Learning
Embargo End Date
Type
Conference proceeding
Date
2025
License
Language
English
Collections
Research Projects
Organizational Units
Journal Issue
Abstract
This paper introduces innovative solutions to enhance spatial controllability in diffusion models reliant on text queries. We first introduce vision guidance as a foundational spatial cue within the perturbed distribution. This significantly refines the search space in a zero-shot paradigm to focus on the image sampling process adhering to the spatial layout conditions. To precisely control the spatial layouts of multiple visual concepts with the employment of vision guidance, we propose a universal framework, Layered Rendering Diffusion (LRDiff), which constructs an image-rendering process with multiple layers, each of which applies the vision guidance to instructively estimate the denoising direction for a single object. Such a layered rendering strategy effectively prevents issues like unintended conceptual blending or mismatches while allowing for more coherent and contextually accurate image synthesis. The proposed method offers a more efficient and accurate means of synthesising images that align with specific layout and contextual requirements. Through experiments, we demonstrate that our method outperforms existing techniques, both quantitatively and qualitatively, in two specific layout-to-image tasks: bounding box-to-image and instance mask-to-image. Furthermore, we extend the proposed framework to enable spatially controllable editing. The project page is available here.
Citation
Z. Qi, G. Huang, C. Liu, and F. Ye, “Layered Rendering Diffusion Model for Controllable Zero-Shot Image Synthesis,” Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) , vol. 15124, pp. 426–443, Jan. 2025, doi: 10.1007/978-3-031-72848-8_25.
Source
Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Conference
European Conference on Computer Vision (ECCV)
Keywords
Controlled image generation, Diffusion Models, Image Editing
Subjects
Source
European Conference on Computer Vision (ECCV)
Publisher
Springer Nature