Photography

Nvidia's new AI takes a one-stop approach to fixing grainy photos

Nvidia's new AI takes a one-st...
The Noise2Noise AI image-enhancing technology was developed by researchers from NVIDIA, MIT and Finland’s Aalto University
The Noise2Noise AI image-enhancing technology was developed by researchers from NVIDIA, MIT and Finland’s Aalto University
View 4 Images
Nvidia's Noise2Noise AI is capable of producing a noise-free image without ever seeing an original clean version
1/4
Nvidia's Noise2Noise AI is capable of producing a noise-free image without ever seeing an original clean version
The Noise2Noise AI image-enhancing technology was developed by researchers from NVIDIA, MIT and Finland’s Aalto University
2/4
The Noise2Noise AI image-enhancing technology was developed by researchers from NVIDIA, MIT and Finland’s Aalto University
The Noise2Noise AI image-enhancing technology was developed by researchers from NVIDIA, MIT and Finland’s Aalto University
3/4
The Noise2Noise AI image-enhancing technology was developed by researchers from NVIDIA, MIT and Finland’s Aalto University
Nvidia's Noise2Noise AI is capable of producing a noise-free image without ever seeing an original clean version
4/4
Nvidia's Noise2Noise AI is capable of producing a noise-free image without ever seeing an original clean version
View gallery - 4 images

If you've ever taken a photo in low light you've probably encountered the grainy effect that can dilute the finished product. A new AI tool could prove an incredibly easy way to remove this so-called noise, with the ability to automatically produce a clean image after analyzing only the corrupted version.

The AI was built by researchers from Nvidia, MIT and Finland's Aalto University, who set out to improve on recent work in the field of machine learning and image processing. This has led to neural networks that can scan a noisy image along with a clean version, and then use machine learning to bridge the gap.

Their solution, dubbed Noise2Noise, is capable of producing a noise-free image without ever seeing an original clean version, removing artifacts, noise and grain automatically. The team developed the tool by training it on a catalog of 50,000 images, and tested it out on three different datasets comprising images of different kinds.

Nvidia's Noise2Noise AI is capable of producing a noise-free image without ever seeing an original clean version
Nvidia's Noise2Noise AI is capable of producing a noise-free image without ever seeing an original clean version

"It is possible to learn to restore signals without ever observing clean ones, at performance sometimes exceeding training using clean exemplars," the researchers stated in their paper. "The neural network is on par with state-of-the-art methods that make use of clean examples – using precisely the same training methodology, and often without appreciable drawbacks in training time or performance."

While this kind of tool could one day make Instagram feeds a lot prettier, in the realm of science it could also have quite a meaningful impact. Medical images taken from by MRI and space images taken with scientific instruments are two typically grainy examples that could benefit from this type of technology.

Nvidia's Noise2Noise AI is capable of producing a noise-free image without ever seeing an original clean version
Nvidia's Noise2Noise AI is capable of producing a noise-free image without ever seeing an original clean version

"There are several real-world situations where obtaining clean training data is difficult: low-light photography (e.g. astronomical imaging), physically-based rendering, and magnetic resonance imaging," the team says. "Our proof-of-concept demonstrations point the way to significant potential benefits in these applications by removing the need for potentially strenuous collection of clean data. Of course, there is no free lunch – we cannot learn to pick up features that are not there in the input data – but this applies equally to training with clean targets."

A research paper describing the technology is available online, with the work to be presented at the International Conference on Machine Learning in Stockholm, Sweden, this week. The video below provides a few more examples of it in action.

Source: Nvidia

Research at NVIDIA: AI Can Now Fix Your Grainy Photos by Only Looking at Grainy Photos

View gallery - 4 images
4 comments
Malatrope
I spent my career investigating and developing image processing algorithms, and I would like to comment here that this approach represents a very dangerous philosophy. It is okay to use a method like this to "clean up" images that are captured in the interest of art or culture, but it is absolutely unacceptable to use it for data collection or medical imaging.
Essentially, this is representing each small region of an image by a guess based on similar datasets. The results will look cleaner, but they do not reflect reality! It is quite easy to literally paint out significant signals within the image. The signatures of small objects are destroyed.
AI-based cleanup filters like this should never be used to transform mission-critical images like medical X-rays or military sensor data.
IvanWashington
WOW! I want this!
toyhouse
The first thing that came to mind was applying the tech to antique photos to clear up grain and other issues - perhaps even focus? Could it also learn to add color or possibly apply 3D - from info it's fed as the article states? Lastly, what about video? Or are these things already happening? Very-very impressive - and spooky at the same time.
Ralf Biernacki
@toyhouse: This is not magic---it will not POOF! repair any sort of image defect, or ABRACADABRA! make flat images into 3D. The approach relies on the known statistical features on random noise, (in conjunction with a big database of the statistical patterns of noiseless images) and it can /only/ reduce noise. It still is impressive, and a great way to rescue low-light images. I fully expect camera makers to pay to license this patent, and incorporate this gimmick in their cameras in lieu of actual sensor improvements. After all, casual users are not likely to understand what Malatrope is pointing out.