Nikon Z9's new sensor may usher in a big change in photography

Nikon recently announced its new flagship camera, the Z9. The camera represents Nikon's first mirrorless camera, fully geared for professional photographers and hybrid shooters, including use in the demanding fields of photojournalism, sports, nature, birding and any other use-cases. There is a huge spec list designed for demanding cameras to shoot with. Incredibly fast at high resolution. The $5,500 Z9 is also Nikon's first camera to avoid a traditional mechanical shutter, which allows it to achieve new levels of speed and autofocus performance.

Fast motion is great, especially for sports photographers. But it's interesting to think about where this technology might be used to take more traditional-style cameras in the future. This could be the first step towards large-format cameras embracing the computational smarts that smartphone cameras have been embracing for years.

Nikon made no mention of things like computational photography or cyclic buffering for HDR-style photos that smartphones capture up to nine or 10 frames simultaneously and combine them with each press of the shutter button. But the new 45.7-megapixel full-frame backside-illuminated stacked CMOS sensor isn't far off against the phone, at least in terms of core design. This type of build uses a sandwich architecture of sensors, logic boards and dedicated RAM - providing incredibly fast readout speeds.

Today, this enables the Z9 to use a full-time electronic shutter with a fastest shutter speed of 1/32,000 of a second and achieve incredibly quick burst shooting. It can capture 20 frames per second in RAW/JPG at full resolution or as fast as 120 fps at 11 megapixels, without any audible sound (optional simulated shutter sound can be enabled for audible cues). The new Exped 7 processor and dual CFexpress/XQD card slots give the Z9 a claimed 1,000-shot buffer at full resolution in high-efficiency compressed Raw, but it's the stacked sensor's fast readout speed that may be the key to the computational photography puzzle . .

As the first of the major camera manufacturers to abandon the mechanical shutter, this puts Nikon ahead of its competitors in the race for computational photography. Sony's A1 and A9 lines already use stacked sensors for faster readout speeds, making electronic shutters viable for full-time duty, and Canon's upcoming R3 will use the same technology. Moving to an entirely electronic shutter has been the logical next development for cameras, although it will be up to Nikon to prove that its electronic shutter is up to the everyday tasks and demands of pro photographers.

To date, camera manufacturers' efforts to implement computational photography have been limited to features such as Olympus' Live ND and Panasonic's post-focus and in-camera focus stacking. Handy features, yes, but these are sideshows compared to the paradigm shift that full computational photography implemented with every press of the shutter could one day be. Om Systems, the newly rebranded Olympus, recently promised to use computational photography technology in its next camera, but we'll have to see if that's the main focus or just another feature.

Deep learning, which is used in the new Z9's object detection autofocus system, has been used to some capacity before from Olympus, Panasonic, and Canon. This autofocus does a good job of improving tracking performance, but in the end, a mirrorless camera still captures a single image that's limited by the sensor's dynamic range.

The primary hurdle most likely lies in the data throughput and image processing pipeline of stacked sensors like the Z9 and other pro- or enthusiast-level mirrorless cameras becoming fully computational. Ten frames captured simultaneously from a 45-megapixel full-frame sensor and combined into one file will be exponentially larger than a similar collection of images taken from a smartphone sensor at a fraction of the size.

Additionally, cyclic buffering is necessary to continuously write and rewrite images in the camera's buffer in the background before the shutter is pressed. The new processor of the Z9 may also not be suitable for these tasks. In the smartphone space, CPUs are fine-tuned for this processing, even sometimes using dedicated hardware, but cameras are not built the same way. It's possible that more innovation is still needed at the CPU level from camera manufacturers.

There are some obvious advantages to using computational photography. Most modern smartphones can create a balanced exposure with well-lit subjects, shadows full of visual detail, and visible clouds – all in a single frame. Advances like Night Sight and Night Mode let you do things that are very difficult to achieve with a standard camera, while Google continues to bring out new computational tricks to keep subjects sharp when they're in motion, and Apple uses computational data. does. continues to do. Also allows RAW files.

On the other hand, a photo taken with today's most advanced mirrorless cameras—which excel at sharpness and resolution—results in making some sacrifices, like blowing out highlights or crushing shadow detail in high-contrast daytime scenes. It seems to require minimal post-processing and editing for most smartphones, ideally from a RAW file that should be exported as a JPG or other universal format. Computational photography coming to dedicated camera systems could revitalize the camera market, though it could see camera makers eventually finding connected Wi-Fi apps that aren't terrible — another tall order, of course.

Cameras like the Z9 could be the bridge of that path, something that can be appreciated even by professional photographers, who can spend less time editing than many of their clients. This could make full-size cameras a little more exciting again, even if it's a "what's the picture?" May blur the lines ahead.