Lumiere

Lumiere Overview

Lumiere is a text-to-video diffusion model developed by Google Research, designed to synthesize videos that portray realistic, diverse and coherent motion. It introduces a Space-Time U-Net architecture that generates the entire temporal duration of the video at once, through a single pass in the model. This is in contrast to existing video models which synthesize distant keyframes followed by temporal super-resolution. Lumiere is capable of a wide range of content creation tasks and video editing applications, including image-to-video, video inpainting, and stylized generation.

Lumiere Highlights

• Lumiere uses a Space-Time U-Net architecture that generates the entire temporal duration of the video at once, improving global temporal consistency.
• It leverages a pre-trained text-to-image diffusion model to directly generate a full-frame-rate, low-resolution video by processing it in multiple space-time scales.
• Lumiere facilitates a wide range of content creation tasks and video editing applications, including image-to-video, video inpainting, and stylized generation.