Google trained its portrait lighting AI with 64-camera and 331-light rig

Our phone cameras are getting better and better, and AI has been all the rage in photography lately. Google wants to bring AI lighting to your photos and help you relight your images after you’ve taken them. To train the algorithm, Google used an impressive rig with 64 cameras and 331 LED lights, and judging from the demos, the results are pretty awesome.

Portrait Light is a new feature recently added to the Pixel Camera and Google Photos apps. It was originally introduced for Pixel 4a 5G and Pixel, but then older Pixel phones also got an access to it. On its AI Blog, Google explains how AI was trained for Portrait Light and how this feature enables you to change the lighting in a photo after it has been taken.

Google had to capture literally millions of pairs of portraits. One photo in the pair had the extra light, and the other didn’t. They shot the photos using the rig I mentioned above, which contains 64 cameras with different viewpoints and 331 individually-programmable LED light sources. Here’s a little peek at what it looks like:

The algorithm trained this way enabled two new capabilities, as Google explains:

• Automatic directional light placement: For a given portrait, the algorithm places a synthetic directional light in the scene consistent with how a photographer would have placed an off-camera light source in the real world.
• Synthetic post-capture relighting: For a given lighting direction and portrait, synthetic light is added in a way that looks realistic and natural.

In other words, you can create lighting as you please on your phone after taking the photo. “We optimized the full pipeline to run at interactive frame-rates on mobile devices, with total model size under 10 MB,” Google writes. And judging from the examples, the results are much more realistic than on my Huawei (my phone does a pretty lame job, actually).

Of course, I’m always into setting or finding the best light possible and not adjusting the light in post. However, if we’re shooting with our phone it probably means we’re on the go. Therefore, we might not be able to really choose the lighting on all occasions. In such cases, it’s pretty useful to have the relighting option that does a good job, and Google’s Portrait Light sure looks like it.

[via Engadget]