This week, Google released Lumiere, which is a space-time diffusion model for video generation. In 2023 we saw AI image generators take off, and many say that 2024 will be the year that AI video generators follow suit.
In this guide, we'll look at what Lumiere is, how it works, and key features of the new model.
What is Google Lumiere?
As mentioned, Google Lumiere uses a space-time diffusion model and the Space-Time U-Net (STUNet) architecture.
This unique architecture sets it apart from existing AI video generation tools like Runway, Pika, and Stability AI, typically generate high-quality video frames. If you've tested out these tools before, you know they often struggle with creating realistic motion over longer durations.
These models usually generate keyframes first and then use temporal super-resolution to fill in the gaps, which can lead to inconsistencies in motion and limitations in video length and quality.
In contrast, Lumiere's STUNet architecture allows for the generation of the entire video in a single pass, resulting in more realistic and coherent motion.
Key Features of Google Lumiere
Technical architecture aside, a few of the key features of Lumiere include:
- Text-to-Video: Lumiere can be used to generate videos from textual inputs that features realistic motion, which has been a challenging aspect in AI video creation.
- Multimodal Capabilities: Lumiere is not just limited to text inputs; it can work with images and potentially other modalities, making it a versatile tool for various applications.
- Advanced Editing Options: The model supports features like video inpainting and cinemagraph creation, pushing the boundaries of creative video editing.
How to use Google Lumiere
At the time of writing, Google hasn't made Lumiere available for public use yet. As of now, we just have the research paper.
As soon as the model becomes available for public use, we'll be sure to test it out and write a getting started tutorial. Until then, you can sign up for our newsletter before to stay up to date with AI.
