Scene Dreamer

SceneDreamer Overview

SceneDreamer is an innovative AI tool developed by Nanyang Technological University that is capable of generating unbounded 3D scenes from 2D image collections. The tool uses a unique learning paradigm that includes an efficient 3D scene representation, a generative scene parameterization, and an effective renderer that leverages knowledge from 2D images. SceneDreamer synthesizes large-scale 3D landscapes from random noises, without the need for any 3D annotations. The tool is capable of creating diverse landscapes across different styles, with 3D consistency, well-defined depth, and free camera trajectory.

SceneDreamer Highlights

• SceneDreamer uses a bird's-eye-view (BEV) representation generated from simplex noise, which includes a height field and a semantic field. This allows for efficient training and a disentangled geometry and semantics.
• The tool employs a novel generative neural hash grid to parameterize the latent space given 3D positions and the scene semantics. This aims to encode generalizable features across scenes and align content.
• SceneDreamer uses a neural volumetric renderer, learned from 2D image collections through adversarial training, to produce photorealistic images.