DreamFusion

DreamFusion Overview

DreamFusion is an innovative tool that uses a text-to-image generative model, Imagen, to optimize a 3D scene based on a given caption. It introduces a unique method called Score Distillation Sampling (SDS) to generate samples from a diffusion model by optimizing a loss function. DreamFusion can optimize samples in any parameter space, including 3D, as long as it can be mapped back to images differentiably. It uses a 3D scene parameterization similar to Neural Radiance Fields (NeRFs) for this differentiable mapping. While SDS alone produces reasonable scene appearance, DreamFusion enhances it with additional regularizers and optimization strategies to improve geometry. The resulting trained NeRFs are coherent, with high-quality normals, surface geometry, depth, and are relightable with a Lambertian shading model.

DreamFusion Highlights

• Uses a unique method called Score Distillation Sampling (SDS) to generate samples from a diffusion model by optimizing a loss function.
• Can optimize samples in any parameter space, including 3D, as long as it can be mapped back to images differentiably.
• Enhances the scene appearance with additional regularizers and optimization strategies to improve geometry.