Authors
Haochen Wang, Xiaodan Du, Jiahao Li, Raymond A. Yeh, Greg Shakhnarovich
TTI-Chicago; Purdue University
Portals
Abstract
A diffusion model learns to predict a vector field of gradients. We propose to apply chain rule on the learned gradients, and back-propagate the score of a diffusion model through the Jacobian of a differentiable renderer, which we instantiate to be a voxel radiance field. This setup aggregates 2D scores at multiple camera viewpoints into a 3D score, and repurposes a pretrained 2D model for 3D data generation. We identify a technical challenge of distribution mismatch that arises in this application, and propose a novel estimation mechanism to resolve it. We run our algorithm on several off-the-shelf diffusion image generative models, including the recently released Stable Diffusion trained on the large-scale LAION dataset.
Contribution
- We propose a method for lifting a 2D diffusion model to 3D via an application of the chain rule
- We illustrate the challenge of OOD when using a pretrained denoiser and propose Perturb-and-Average Scoring to resolve it
- We point out the subtleties and open problems on applying Perturb-and-Average Scoring as gradient for optimization
- We demonstrate the effectiveness of SJC for the task of 3D text-driven generation
Related Works
Neural radiance fields (NeRF); 2D-supervised 3D GANs; CLIP-guided, optimization-based 3D generative models; DreamFusion