DIY Photography

Your one stop shop for everything photo-video

  • News
  • Inspiration
  • Reviews
  • Tutorials
  • DIY
  • Gear
Search

Submit A Story

Nvidia’s “content-aware” cloning tools use deep learning AI for incredibly realistic results

Apr 24, 2018 by John Aldred 4 Comments

  • Share
  • Tweet
  • Flipboard
  • WhatsApp

When the Content-Aware Fill tools were added to Photoshop a number of years back, they were hailed as being the best thing since Photoshop itself was created. Now, with a couple of clicks, you could get rid of the stuff you didn’t want in your image and Photoshop would magically replace it with what you wanted. The reality was that it didn’t always do what it said on the tin, leading to the nickname “Content-Aware Fail”.

It’s come a long way since then. It’s smarter than it was, and soon it may expand to include fill sources from Adobe’s vast stock library. Nvidia’s taking a slightly different approach, though, using deep learning AI to help fill in the gaps, and while the aesthetics are still fairly primitive, the AI seems to do a great job of recognising what’s what.

YouTube video

The artifacts in the final result sometimes seem quite obvious. It tends to not do very well with eyes, for example. At least from this demo. But the potential to actually recognise what different parts of the face are, and intelligently fill in the content based of knowing “This is a hairline” or “This is the bridge of the nose” is quite incredible.

Some of the less “organic” shots, like building interiors look like they struggle a little more with this artifacting than human faces. More than likely, this is simply due to being fed more organic looking outdoor landscapes and portraits. So it’s not entirely sure how to deal with these slightly-but-n0t-quite-straight lines of interiors.

Even though it’s far from perfect, it’s a fantastic proof of concept that it can recognise what something is in an image, and then generate entirely new pixels to fill in the gaps. There’s no word on whether this will ever come to a final product we can download and use, but that it exists shows the way things are going, and Nvidia can’t be the only ones working on something like this.

And it’s a learning AI. So, it will get better over time as it sees more images. A paper published by the team who created the concept includes some excellent examples showing what can be done when it’s well trained. The comparison below shows Nvidia’s method vs Adobe’s. The image with the white blocks is what was fed into each application, the two in the middle are Adobe & Nvidia’s results, and on the right is the original image.

You can see that the results from Nvidia’s method are clearly well ahead of Adobe’s. But, I’m sure Adobe have been working on something similar behind the scenes, especially since they started combining their content-aware tools with the Adobe Stock library.

While technology such as this would obviously become a defacto replacement for the current Content Aware tools, I could easily see this becoming a favourite for those restoring old photographs. When grandma finds those decades-old prints in the attic and wants you to scan them in and clean up all the cracks and gaps, you’d be able to do it with relative ease using something like this.

Now we just have to wait and see who can get something released first.

FIND THIS INTERESTING? SHARE IT WITH YOUR FRIENDS!

  • Share
  • Tweet
  • Flipboard
  • WhatsApp

Related posts:

Adobe’s new Content-Aware Fill interface makes it really easy to fake photos Adobe’s new deep learning tech might make green screen keying obsolete Content Aware Crop is one of the features coming in a major Photoshop CC Update Adobe’s new “Cloak” technology is like content aware fill for video

Filed Under: news Tagged With: Adobe, Adobe Photoshop, Nvidia, retouching

John Aldred: from diyphotography.net

About John Aldred

John Aldred is a photographer with over 20 years of experience in the portrait and commercial worlds. He is based in Scotland and has been an early adopter - and occasional beta tester - of almost every digital imaging technology in that time. As well as his creative visual work, John uses 3D printing, electronics and programming to create his own photography and filmmaking tools and consults for a number of brands across the industry.

« Photographer shows the real impact of the clothing industry on the environment
This is the world’s first Sony E-Mount 35mm film camera – and it’s open source »

Submit A Story

Get our FREE Lighting Book

DIYP lighting book cover

* download requires newsletter signup

Recent Comments

Free Resources

Advanced lighting book

Learn photography

Recent Posts

  • SmallRig x Mikevisuals Tracking POV Kit is an interesting if niche camera rig
  • Modern-day Snow White wins International Wedding Photographer of the Year
  • The slow death of creativity in modern photography
  • Holiday Gift Guide – Our favourite gadgets and accessories of 2023
  • Stability AI’s new AI generator creates images faster than you imagine them

Udi Tirosh: from diyphotography.netUdi Tirosh is an entrepreneur, photography inventor, journalist, educator, and writer based in Israel. With over 25 years of experience in the photo-video industry, Udi has built and sold several photography-related brands. Udi has a double degree in mass media communications and computer science.

Alex Baker: from diyphotography.netAlex Baker is a portrait and lifestyle driven photographer based in Valencia, Spain. She works on a range of projects from commercial to fine art and has had work featured in publications such as The Daily Mail, Conde Nast Traveller and El Mundo, and has exhibited work across Europe

David Williams: from diyphotography.netDave Williams is an accomplished travel photographer, writer, and best-selling author from the UK. He is also a photography educator and published Aurora expert. Dave has traveled extensively in recent years, capturing stunning images from around the world in a modified van. His work has been featured in various publications and he has worked with notable brands such as Skoda, EE, Boeing, Huawei, Microsoft, BMW, Conde Nast, Electronic Arts, Discovery, BBC, The Guardian, ESPN, NBC, and many others.

John Aldred: from diyphotography.netJohn Aldred is a photographer with over 20 years of experience in the portrait and commercial worlds. He is based in Scotland and has been an early adopter - and occasional beta tester - of almost every digital imaging technology in that time. As well as his creative visual work, John uses 3D printing, electronics and programming to create his own photography and filmmaking tools and consults for a number of brands across the industry.

Dunja Djudjic: from diyphotography.netDunja Djudjic is a multi-talented artist based in Novi Sad, Serbia. With 15 years of experience as a photographer, she specializes in capturing the beauty of nature, travel, and fine art. In addition to her photography, Dunja also expresses her creativity through writing, embroidery, and jewelry making.

Copyright © DIYPhotography 2006 - 2023 | About | Contact | Advertise | Write for DIYP | Full Disclosure | Privacy Policy