Smartphones have revolutionized photography, making it vastly easier than ever before to capture and share great images. And on the other side of the coin, camera advances are probably responsible for a lot of smartphone sales these days; phones have been more than good enough in most areas for many years now, and a terrific new camera provides a compelling reason to upgrade where a processing boost or RAM upgrade might not move the needle.
And they do keep getting better, thanks to a number of creative software and hardware developments that keep pushing super-compact imaging to new heights. Most people don't know or care how they're getting better; they don't need to understand. But the technically-minded will find some interest in something like the new Samsung Isocell GN2 sensor, currently in mass production, which pushes forward the state of the art with some clever ideas.
At the most basic level, it's a smartphone camera sensor with a monster 50-megapixel (8,160 x 6,144) resolution, far beyond what most of us will ever need unless we're cropping way in to look at fine details. Likewise, it can shoot video in massive 8K resolution, despite there not being all that many 8K screens to play it back on. Switch to 1080p HD, and it's so fast it can give you super-slow motion recorded at 480 fps, stretching time by a factor of 20 if you're watching it back at a cinematic 24 fps.
If 50 megapixels isn't enough for you, it can super-resolve the image "using an intelligent re-mosaic algorithm, creating three individual layers of 50Mp frames in green, red and blue. These frames are then up-scaled and merged to produce a single ultra-high 100Mp resolution photograph." It can perform this operation at an astonishing rate: 10 times per second. May your storage drives rest in peace.
In low light situations, it pretends to be a lower-resolution sensor with larger pixels, combining every four pixels into a single "bin" to soak up as much light as possible without introducing extra noise by jacking up the ISO sensitivity.
And from there, things get really fun.
Dual Pixel Pro: turning every pixel into a phase detect AF point
You've probably heard the term "phase detect" used to describe autofocus points. In typical camera sensors, what this means is that at strategic points around the sensor, special autofocus pixels are placed in the sensor matrix, with their right or left side blocked off so they pick up light from only one side of the lens. In this way, the images from right- and left-looking phase detect pixels can be compared, and the camera's processor can use phase differences to calculate the focus distance and make the necessary adjustments to bring the desired subjects into sharp focus.
The more of these phase detect pixels you put in your sensor, the greater the number of points the camera can use to build its understanding of the image and the faster it can focus, generally speaking. So phase detect points are exploding in numbers; the latest Sony flagship full-frame Alpha 1 will have 759 of them dotted through its 50-megapixel sensor, and it'll surely be a monster of a focus system.
The drawback, particularly for compact smartphone sensors using tiny pixels, is that every pixel devoted to phase detection is a pixel that's not drawing in light or color information, says Samsung, meaning the image suffers. So it's come up with a "Dual Pixel Pro" autofocus system that uses every pixel on the sensor as a phase detect point, while upgrading the concept to search for phase differences diagonally and vertically as well as just left/right horizontally.
To achieve this, each of the 50 million-odd pixels on the Isocell GN2 sensor has not one but two photodiodes in it. In the previous Samsung Dual Pixel system, these were uniformly divided into left and right looking diodes. But Samsung found that while this improved image quality and autofocus, there were still issues, particularly with patterns such as horizontal stripes.
So the Dual Pixel Pro system uses standard left/right splits for the red and blue pixels, but splits the green ones diagonally, giving the processor vertical phase shift information to work with as well as horizontal. Samsung says this gives the Isocell GN2 super-fast and accurate autofocus capabilities even in low light or with fast-moving subjects, while taking in light at every pixel for optimum image quality. Check out a video:
Staggered HDR and Smart ISO Pro
HDR, or High Dynamic Range photography, seeks to address one of the failings of camera sensors, which is their limited dynamic range, or the difference in illumination between the brightest and darkest points they can capture in a single image. Canny photographers have been coming up with clever ways around this limitation for nearly 170 years, believe it or not. Check out this HDR image Gustave Le Gray created back in 1856, combining two exposures to give outstanding detail in both the bright sky and the dark ocean.
These days, of course, it's a lot easier. Most smartphones will automatically take several images and combine them to produce an image with a higher dynamic range than the sensor can do by itself. You never need to think about it, unless you want to fiddle with a slider somewhere and turn your image into a clownish, glowing monstrosity.
Samsung is achieving HDR in two ways with the new sensor. The first is staggered HDR, which uses rolling shutters over the same pixel arrays to capture short, middle and long exposures of the same scene. The key benefit here is in energy consumption; Staggered HDR uses nearly a quarter less energy than Samsung's previous real-time HDR mode.
Then there's Smart ISO Pro, which introduces dual conversion gain to Samsung's imaging lineup. This comes down to how the sensor reads the charge in each pixel's two photodiodes to work out what it's seeing. Dual conversion gain reads each photodiode's output twice, with two gain factors applied, thus effectively grabbing higher and lower ISO versions of each single shot. This means you can extend the camera's dynamic range without affecting your signal-to-noise ratio even before you start taking multiple exposures. The benefit is that you eliminate the ghosting and artefacts that can be introduced in multiple-exposure HDR images if a subject moves between exposures.
Smart ISO Pro also helps with low-light photography, shooting multiple images in the dark at high ISO and combining them all to deliver a single image Samsung says is close to 1 million ISO. We've seen this approach before with Google's Night Sight.
This sure looks like a pretty neat smartphone camera sensor; Samsung's cameras have been right up there with the best of Apple and Google for many years now, and while a lot of it will come down to how a company's camera software uses what it gets from the sensor, a high-tech upgrade like this certainly gives the image processors more and better information to start with.
Samsung says the Isocell GN2 sensor is already in mass production, and while nobody's officially announced a phone that uses it, Samsung is a major supplier for many brands. Leaks suggest the Xiaomi Mi 11 Ultra will be one of the first phones to get the sensor.
As we said at the top, the vast majority of smartphone buyers will never know or care what's going on under the hood, but I find it fascinating to see some of the work that goes into the ever-escalating smartphone camera war. We all get to enjoy the benefits as our cameras get smarter!
Source: Samsung
