Nvidia’s “content-aware” cloning tools use deep learning AI for incredibly realistic results

Apr 24, 2018

John Aldred

John Aldred is a photographer with over 20 years of experience in the portrait and commercial worlds. He is based in Scotland and has been an early adopter – and occasional beta tester – of almost every digital imaging technology in that time. As well as his creative visual work, John uses 3D printing, electronics and programming to create his own photography and filmmaking tools and consults for a number of brands across the industry.

Nvidia’s “content-aware” cloning tools use deep learning AI for incredibly realistic results

Apr 24, 2018

John Aldred

John Aldred is a photographer with over 20 years of experience in the portrait and commercial worlds. He is based in Scotland and has been an early adopter – and occasional beta tester – of almost every digital imaging technology in that time. As well as his creative visual work, John uses 3D printing, electronics and programming to create his own photography and filmmaking tools and consults for a number of brands across the industry.

Join the Discussion

Share on:

When the Content-Aware Fill tools were added to Photoshop a number of years back, they were hailed as being the best thing since Photoshop itself was created. Now, with a couple of clicks, you could get rid of the stuff you didn’t want in your image and Photoshop would magically replace it with what you wanted. The reality was that it didn’t always do what it said on the tin, leading to the nickname “Content-Aware Fail”.

It’s come a long way since then. It’s smarter than it was, and soon it may expand to include fill sources from Adobe’s vast stock library. Nvidia’s taking a slightly different approach, though, using deep learning AI to help fill in the gaps, and while the aesthetics are still fairly primitive, the AI seems to do a great job of recognising what’s what.

YouTube video

The artifacts in the final result sometimes seem quite obvious. It tends to not do very well with eyes, for example. At least from this demo. But the potential to actually recognise what different parts of the face are, and intelligently fill in the content based of knowing “This is a hairline” or “This is the bridge of the nose” is quite incredible.

Some of the less “organic” shots, like building interiors look like they struggle a little more with this artifacting than human faces. More than likely, this is simply due to being fed more organic looking outdoor landscapes and portraits. So it’s not entirely sure how to deal with these slightly-but-n0t-quite-straight lines of interiors.

Even though it’s far from perfect, it’s a fantastic proof of concept that it can recognise what something is in an image, and then generate entirely new pixels to fill in the gaps. There’s no word on whether this will ever come to a final product we can download and use, but that it exists shows the way things are going, and Nvidia can’t be the only ones working on something like this.

And it’s a learning AI. So, it will get better over time as it sees more images. A paper published by the team who created the concept includes some excellent examples showing what can be done when it’s well trained. The comparison below shows Nvidia’s method vs Adobe’s. The image with the white blocks is what was fed into each application, the two in the middle are Adobe & Nvidia’s results, and on the right is the original image.

You can see that the results from Nvidia’s method are clearly well ahead of Adobe’s. But, I’m sure Adobe have been working on something similar behind the scenes, especially since they started combining their content-aware tools with the Adobe Stock library.

While technology such as this would obviously become a defacto replacement for the current Content Aware tools, I could easily see this becoming a favourite for those restoring old photographs. When grandma finds those decades-old prints in the attic and wants you to scan them in and clean up all the cracks and gaps, you’d be able to do it with relative ease using something like this.

Now we just have to wait and see who can get something released first.

Filed Under:

Tagged With:

Find this interesting? Share it with your friends!

John Aldred

John Aldred

John Aldred is a photographer with over 20 years of experience in the portrait and commercial worlds. He is based in Scotland and has been an early adopter – and occasional beta tester – of almost every digital imaging technology in that time. As well as his creative visual work, John uses 3D printing, electronics and programming to create his own photography and filmmaking tools and consults for a number of brands across the industry.

Join the Discussion

DIYP Comment Policy
Be nice, be on-topic, no personal information or flames.

Leave a Reply

Your email address will not be published. Required fields are marked *

4 responses to “Nvidia’s “content-aware” cloning tools use deep learning AI for incredibly realistic results”

  1. Mark Niebauer Avatar
    Mark Niebauer

    Photoshop and other programs have had content aware software for some years now. I wonder if this will make it better?

  2. Henry Rodgers Avatar
    Henry Rodgers

    Retouching the skin of elderly people is not advised. (or attractive for that matter) I learned the hard way.

  3. David Fiorito Avatar
    David Fiorito

    This is why I shoot film and take the image I get as it is.

  4. Rick Avatar
    Rick

    Totally stripped the character out of Ernest Borgnine’s face. Some things just shouldn’t be messed with.