We've come a long way from Will Smith eating spaghetti in the last 10 months. AI-generated video is advancing at a jaw-dropping rate – and Google's extraordinary new space-time diffusion model Lumiere shifts the goalposts yet again.
Lumiere can create remarkably realistic – or high quality surrealistic – video clips up to five seconds in length. It can animate still images, or just portions of them, in response to natural language text prompts about what you'd like to see.
It can take an image, clone the style of that image, and then use that style to create a bunch of videos on other topics that look and feel so similar they could've come out of a branding agency.
It can take your own source video, and turn everything into Lego, or origami, or flowers – you just have to tell it to.
And if the demos above are any indication, it has by far the most advanced video inpainting capabilities we've ever seen. You can simply paint over a part of the image you don't like, and Lumiere will auto-fill that area so beautifully that you'd likely not even notice if you weren't looking for it. Ex-boyfriend in your favorite video? Not for long.
The research team involved says Lumiere's "space-time U-net architecture" builds the entire length of the video at once, in a single pass – as opposed to previous models, which would often generate a start and an end frame, then try to guess what would happen in between.
However it's done, the results speak for themselves – this is the new state of the art in generative AI video, it's frankly staggering, and it'll probably look as goofy and crappy as Will Smith eating spaghetti within a few months ... Just in time for the next US Presidential election. Yippee.
For now, it's just a research project – which saves Google from having to aggressively neuter the system in service of copyright, misinformation, safety, hate speech, nudity, privacy and all manner of other policies – a process which invariably leads to lower-quality output in these generative models.
But it's still an enormous leap forward, and it'll be fascinating to see how well Lumiere works if and when we, the unwashed and cheeky masses, get our hands on it.
Source: Google Research
