Nvidia’s “content-aware” cloning tools use deep learning AI for incredibly realistic results

When the Content-Aware Fill tools were added to Photoshop a number of years back, they were hailed as being the best thing since Photoshop itself was created. Now, with a couple of clicks, you could get rid of the stuff you didn’t want in your image and Photoshop would magically replace it with what you wanted. The reality was that it didn’t always do what it said on the tin, leading to the nickname “Content-Aware Fail”.

It’s come a long way since then. It’s smarter than it was, and soon it may expand to include fill sources from Adobe’s vast stock library. Nvidia’s taking a slightly different approach, though, using deep learning AI to help fill in the gaps, and while the aesthetics are still fairly primitive, the AI seems to do a great job of recognising what’s what.

The artifacts in the final result sometimes seem quite obvious. It tends to not do very well with eyes, for example. At least from this demo. But the potential to actually recognise what different parts of the face are, and intelligently fill in the content based of knowing “This is a hairline” or “This is the bridge of the nose” is quite incredible.

Some of the less “organic” shots, like building interiors look like they struggle a little more with this artifacting than human faces. More than likely, this is simply due to being fed more organic looking outdoor landscapes and portraits. So it’s not entirely sure how to deal with these slightly-but-n0t-quite-straight lines of interiors.

Even though it’s far from perfect, it’s a fantastic proof of concept that it can recognise what something is in an image, and then generate entirely new pixels to fill in the gaps. There’s no word on whether this will ever come to a final product we can download and use, but that it exists shows the way things are going, and Nvidia can’t be the only ones working on something like this.

And it’s a learning AI. So, it will get better over time as it sees more images. A paper published by the team who created the concept includes some excellent examples showing what can be done when it’s well trained. The comparison below shows Nvidia’s method vs Adobe’s. The image with the white blocks is what was fed into each application, the two in the middle are Adobe & Nvidia’s results, and on the right is the original image.

You can see that the results from Nvidia’s method are clearly well ahead of Adobe’s. But, I’m sure Adobe have been working on something similar behind the scenes, especially since they started combining their content-aware tools with the Adobe Stock library.

While technology such as this would obviously become a defacto replacement for the current Content Aware tools, I could easily see this becoming a favourite for those restoring old photographs. When grandma finds those decades-old prints in the attic and wants you to scan them in and clean up all the cracks and gaps, you’d be able to do it with relative ease using something like this.

Now we just have to wait and see who can get something released first.