Adobe’s new “Cloak” technology is like content aware fill for video

The Adobe MAX conference just ended today, and its closing brought about some very cool sneak peeks of Adobe tech. One such piece of new technology is what Adobe call “Cloak”. It allows you to remove unwanted things from videos. Essentially, it’s like Content Aware Fill for video. And it’s pretty amazing.

Adobe do warn, though this is an early look at the technology. It may or may not be features that they integrate into products down the road. But I can’t imagine that this won’t be added into After Effects (in which the tech was demonstrated) or Premiere somewhere down the line.

This kind of thing can “sort of” be done right now with a whole lot of work. You’ve got camera tracking, masking, copying and pasting and painting on multiple layers. You might even need to recreate the entire scene in 3D to fake it in something like Blender. And even then, unless you work for a visual effects house, the time and computing power for complex scenes just isn’t realistic.

Cloak, however, seems to make light work of the lamp post in the scene above. Now, in the previous example, the missing information is there in other frames. It’s relatively straightforward to figure out what should go where given the capabilities of existing software. What is very impressive, though, is this next example.

Here, Cloak is used to remove the chest strap from the guy’s backpack. The application never sees what’s actually underneath this strap, but manages to get rid of it seamlessly.

If you look extremely closely, there is a little artifacting, but it’s very subtle. No doubt the higher resolution the footage, the more apparent this becomes. But, it’s still very early days for this technology. No doubt by the time it hits an official release, assuming it does, they’ll have cleaned up these kinds of issues.

As impressive as this is, I think it’ll be most effective on the little things. Removing the tripod or selfie stick from 360 video footage. Getting rid of dust on your sensor or specs backlit scratches on your lens filters.

But, impressive it is. This is the kind of feature that may finally see me make the jump from CS6 to CC.