There are now two ways of creating digital images with a camera. You can either follow a software-centric computational photography approach. The other way is to stick to traditional hardware-centric optical photography. The former is used with AI to help enhance the final image, the latter relies on the quality of the camera’s components (e.g. lens, sensor). The two techniques may differ, but they are not at all on a collision course. They can complement each other and even address each technique’s limitations.

Cameras for the most part are optical devices that create images from the use of a lens. Since the rise of computational photography, new apps have been developed that help create stunning images. The majority of these apps work only with smartphone cameras and that is rightfully so. Due to their smaller size, they do not have the same components used in cameras to create the best images. This is where software comes into play.

The Traditional Camera

The most basic digital camera components are the body, sensor, and lens. You need these basic components to capture images with a camera. Camera manufacturers like Canon and Nikon are established leaders in the camera industry. They focus mainly on DSLR and video cameras, and moving toward mirrorless as the market shifts. Sony has emerged as a leader in mirrorless because they also build sensors and chip components for cameras. For image quality though, nothing can beat having the best lens and sensor combination.

When the first digital cameras were released, they did not have the advanced electronic features available in today’s models. Despite those limitations, they were still capable of taking high-quality images like film cameras. Imaging quality continued to improve with larger sensors, higher resolution, and image stabilization. It can all be implemented in the camera’s hardware, so why would there be a need for software?

Computational Imaging

The rise of computational photography or imaging, was the result of a camera system’s limitation in the hardware. The aftermath of the AI Winter led to an explosion of innovation that followed. This would lead to the integration of software techniques that use AI with imaging systems like cameras, scanners, and copy machines (i.e. sensor-based imaging devices).

The digital camera works best with software since it uses a digital sensor. Cameras today use an image processing unit or chip, that handles the RAW image output to storage. Typically, the image processors in most DSLR and mirrorless cameras do not perform any post retouching on the image. With smartphone cameras, the image processing is done in-camera with software.

The main reason that more software processing on images is required on smartphone cameras is because they are not built the same as traditional cameras. Smartphone cameras have smaller lens, sensor and body. They lack the specifications that would give them the same features as a DSLR. For that reason, computational imaging with the use of software allows smartphone camera apps to create effects that are typical in traditional cameras. This includes features like shallow depth-of-field (DoF) aka bokeh effect and vibration reduction.

Use Cases

Google’s Pixel smartphone camera is a perfect example of the use of computational photography. The stunning imagery on the Pixel 2 was not a result of optics alone, but clever AI. Many of Google’s competitors made use of multiple cameras (dual) during this time, to allow their smartphone cameras to produce similar results to a DSLR to get shallow DoF. What Google lacks in camera hardware, it makes up for with its expertise in software engineering. The Pixel 2 uses what is called Dual Pixel Autofocus (DPAF) which is based on Phase-Detect Auto-Focus (PDAF).

Dual pixel allows the Pixel 2 to use just a single camera to create DoF. This creates what is called Synthetic Depth-of-Field (explained in Google’s blog). According to Google’s explanation of dual pixel:

“If one imagines splitting the (tiny) lens of the phone’s rear-facing camera into two halves, the view of the world as seen through the left side of the lens and the view through the right side are slightly different. These two viewpoints are less than 1 mm apart (roughly the diameter of the lens), but they’re different enough to compute stereo and produce a depth map.”

What Google does is divide the image’s pixels into two diodes which views an image in terms of a left and right side. Algorithms in the software then combine the view from the diodes to produce a stereo image. Google calls its computational photography technique HDR+. This process begins with the image generation after capture. It then undergoes an ML segmentation process to identify the image into layers which is used to separate foreground from the background. A depth map of the layers is then created to finally render the final image.

DSLR also use PDAF (PetaPixel explains that in their article). It is used for faster autofocus not only for stills, but also for video. It just is not as software-centric as the Pixel, but more hardware-centric in its implementation.

On the opposite end, Huawei is adopting a hardware-centric approach along with software in the Mate30 Pro camera. The design was meant to not only make it look like a real camera, but to provide it the functionality as well. As reported by Android Authority, “Huawei has opted to throw more hardware at the camera problem.”

Synopsis

There is no conflict between computational and traditional photography. They can complement each other in ways to improve imaging quality. The use of smartphone cameras still require optics. Normally cameras used on smartphones would produce crappy images because they have smaller sensor sizes and inferior lens quality. The glass is just not as comparable to the type used on let’s say full-frame DSLR. However, applying advanced AI on-chip and software gives it computational photography features that allow it to create stunning images similar to a DSLR.

Computational photography can help process images in real-time in-camera. Apple and other smartphone vendors are incorporating NPU (Neural Processing Units) chips to accelerate image processing and make use of AI-enhanced features. This allows a person of any skill level in photography to capture amazing images. This still requires the optics though, so a basic camera is still needed on the smartphone. Traditional photography relies on the basic components, which have different types. All lenses, for example, are not the same. There are prime lenses, telephoto lenses, and wide-angle lenses used for different types of photography. Each lens also has a different quality from other lenses based on the build of the glass and its focusing features. Likewise, sensor types also vary from cropped to full-frame to medium format.

Ideally, putting computational imaging features on the best DSLR or mirrorless camera should produce the best quality images. It won’t be as practical though, since that will definitely increase costs of both production and retail. It is also not necessary in some cases since the software is mainly used to take the place of hardware. For automated enhancing or retouching the image, software would be more realistic to implement on DSLR and mirrorless cameras.

The reason we don’t see too much of this is due to the market. Established camera vendors follow a different process than smartphone vendors when it comes to making cameras. Camera vendors won’t suddenly make cameras that are more software-centric since they already have the expertise in hardware to deliver the required features. Cameras will continue to evolve, and hopefully we will see more combinations of each technique in retail versions, whether it is a smartphone camera or not.

About the Author

Vincent Tabora is an engineer, photographer, and writer. He’s an editor of High-Definition Pro, where you can read more of his interesting articles. This article was also published here and shared with permission.