Researchers from Nvidia have revealed an impressive new system that uses a deep-learning neural network to effectively create smooth high-quality slow motion videos from footage shot at a regular, low frame rate.
The researchers say that while existing methods to artificially generate slow motion footage from pre-existing video is somewhat effective, it is limited by how many intermediate frames it needs to generate. So, for example, turning a 30-fps video into a half-speed 60-fps video simply requires one extra frame to be generated in between each recorded frame.
But if one wants to generate a 240-fps video from 30-fps footage, that requires seven new frames in between any two consecutive images, and it is here that existing techniques begin to struggle.
To overcome this problem the researchers developed a convolutional neural network trained on over 11,000 various videos that were shot at 240-fps. This allowed the system to accurately learn the optical flow between any two given images and then effectively predict what intermediate images need to be generated to create a flow between the two.
"Our method can generate multiple intermediate frames that are spatially and temporally coherent," the researchers report. "Our multi-frame approach consistently outperforms state-of-the-art single frame methods."
The demonstration video, seen below, is undeniably impressive, with several input videos, ranging from 30-fps to 60-fps, slowed down to super slow-motion rates up to 480-fps. The video also demonstrates the versatility of the system by taking footage already recorded in high-frame-rate slow-motion and slowing it down even further.
While newer smartphones, such as the Samsung Galaxy S9, have the capacity to shoot short videos in slow-motion at high-frame-rates, the researchers suggest this isn't a very practical approach for small mobile devices.
"While it is possible to take 240-frame-per-second videos with a cell phone, recording everything at high frame rates is impractical, as it requires large memories and is power-intensive for mobile devices," the researchers write.
This new system offers exciting new possibilities for everyone from professional filmmakers to professional instagrammers. At this stage the technology is still very much in its prototype phase and the researchers note that it may be currently too processing intensive to easily slip into a smartphone.
"The processing power required for doing this is more than a what a phone would have in this point in time," says Jan Kautz, one of the Nvidia researchers in an interview with ZDnet, "but you could imagine uploading [video] to a server – there are ways of making it work and giving it to users."
The new research will be presented this week at the Computer Vision and Pattern Recognition (CVPR) conference in Salt Lake City, Utah.
Source: Nvidia
