Computers

Nvidia AI-based system transforms any video into fluid super slow motion

The system can artificially generate intermediate frames to transform any video into slow-motion footage
The system can artificially generate intermediate frames to transform any video into slow-motion footage
View 3 Images
The system can effectively turn a 30- or 60-fps video into whatever frame-rate you require
1/3
The system can effectively turn a 30- or 60-fps video into whatever frame-rate you require
The neural network was trained on thousands of slow-motion videos 
2/3
The neural network was trained on thousands of slow-motion videos 
The system can artificially generate intermediate frames to transform any video into slow-motion footage
3/3
The system can artificially generate intermediate frames to transform any video into slow-motion footage

Researchers from Nvidia have revealed an impressive new system that uses a deep-learning neural network to effectively create smooth high-quality slow motion videos from footage shot at a regular, low frame rate.

The researchers say that while existing methods to artificially generate slow motion footage from pre-existing video is somewhat effective, it is limited by how many intermediate frames it needs to generate. So, for example, turning a 30-fps video into a half-speed 60-fps video simply requires one extra frame to be generated in between each recorded frame.

But if one wants to generate a 240-fps video from 30-fps footage, that requires seven new frames in between any two consecutive images, and it is here that existing techniques begin to struggle.

The neural network was trained on thousands of slow-motion videos 
The neural network was trained on thousands of slow-motion videos 

To overcome this problem the researchers developed a convolutional neural network trained on over 11,000 various videos that were shot at 240-fps. This allowed the system to accurately learn the optical flow between any two given images and then effectively predict what intermediate images need to be generated to create a flow between the two.

"Our method can generate multiple intermediate frames that are spatially and temporally coherent," the researchers report. "Our multi-frame approach consistently outperforms state-of-the-art single frame methods."

The demonstration video, seen below, is undeniably impressive, with several input videos, ranging from 30-fps to 60-fps, slowed down to super slow-motion rates up to 480-fps. The video also demonstrates the versatility of the system by taking footage already recorded in high-frame-rate slow-motion and slowing it down even further.

Research at NVIDIA: Transforming Standard Video Into Slow Motion with AI

While newer smartphones, such as the Samsung Galaxy S9, have the capacity to shoot short videos in slow-motion at high-frame-rates, the researchers suggest this isn't a very practical approach for small mobile devices.

"While it is possible to take 240-frame-per-second videos with a cell phone, recording everything at high frame rates is impractical, as it requires large memories and is power-intensive for mobile devices," the researchers write.

The system can effectively turn a 30- or 60-fps video into whatever frame-rate you require
The system can effectively turn a 30- or 60-fps video into whatever frame-rate you require

This new system offers exciting new possibilities for everyone from professional filmmakers to professional instagrammers. At this stage the technology is still very much in its prototype phase and the researchers note that it may be currently too processing intensive to easily slip into a smartphone.

"The processing power required for doing this is more than a what a phone would have in this point in time," says Jan Kautz, one of the Nvidia researchers in an interview with ZDnet, "but you could imagine uploading [video] to a server – there are ways of making it work and giving it to users."

The new research will be presented this week at the Computer Vision and Pattern Recognition (CVPR) conference in Salt Lake City, Utah.

Source: Nvidia

4 comments
Joshua Tulberg
Pretty impressive. I believe GoPro offers something similar in their free software (And I believe it can add more than 1 "faked-tweener" frame as well). The GoPro solution suffers slightly from ghosting effects when the background (behind the moving object) isn't simple. This Nvidia solution seems to suffer slightly from that too, but not very noticeably. I'd like to see a side-by-side comparison (and what Nvidia plans on doing with it).
joshua13
Joshua Tulberg, you are correct, except that Gopro is no longer supporting the "Gopro Studio" software. Adobe and Sony have plug-ins that can also do the fake tweening (a term from animation btw for the people that draw the frames between the key frames) This is defiantly a superior guess as to what the in between frames should look like with out offering much in the way of the uncanny valley that occurs when studios do the slow in CG.
guzmanchinky
Amazing! I film in 4k 60fps but at best you can slow it down by half to get smooth "slow" motion. Ish. I would love to see this software as a plugin to my Adobe Premiere.
DavidB
It'll be interesting the first time someone shows one of these slow-motion videos in court to prove what happened in some incident or a fan shows one to the referees to prove that a player's foot stayed within bounds. This falls into the same category as software that can be used to make it look like someone is saying something they never actually said. Remember when photographs and especially videos were virtually irrefutable evidence?