This insane smartphone app lets you copy and paste elements of the real world into Photoshop

This is one of the craziest things I’ve ever seen, and I had to check my calendar to make sure it wasn’t April 1st, but French artist, designer and programmer, Cyril Diagne has figured out how to use augmented reality to literally copy and paste the world around you straight into a Photoshop document. Yeah, for real. And he’s even released the source code on GitHub!

Diagne announced his creation to the world via Twitter this week, and it really looks like some kind of After Effects fakery. If he hadn’t released the source code, then you probably wouldn’t even believe it was real at all. But it shows him clearly snapping objects around him, which are then cut out from their backgrounds and then pasted right into the current Photoshop document.

4/10 – Cut & paste your surroundings to Photoshop

Code: https://t.co/cVddH3u3ik

Book: @HOLOmagazine
Garment: SS17 by @thekarentopacio
Type: Sainte Colombe by @MinetYoann @ProductionType
Technical Insights: ↓#ML #AR #AI #AIUX #Adobe #Photoshop pic.twitter.com/LkTBe0t0rF

— Cyril Diagne (@cyrildiagne) May 3, 2020

According to Diagne, the “secret sauce” top the augmented reality is BASNet for object detection and background removal – which he describes as “stunning”. OpenCV SIFT  is used to determine where on the computer screen the phone is pointing in order to paste. Then it sends it over to Photoshop via a Python plugin and up it pops in your document.

The latency for picking out the object in the real world, Diagne says, is about 2.5 seconds and it takes about a further 4 seconds in order to paste it into Photoshop. but to even do it that quickly is absolutely incredible.

It works so well, that even Adobe were impressed, with the official Photoshop Twitter account sharing it out.

And again, the OpenCV SIFT trick to find where the phone is pointing at the screen.

I also packaged it as a small python library: https://t.co/en0EyGSklp

Send a camera image + a screenshot and you get accurate x, y screen coordinates! pic.twitter.com/OOCFrrZseZ

— Cyril Diagne (@cyrildiagne) May 3, 2020

It is a fascinating technology. It’s one that will make a lot of people extremely happy, and quite a few people afraid for their job’s future! But it’s not ready for prime time yet. It’s a “research prototype” – essentially a proof of concept. But with the source code freely available (and knowing that Adobe has seen it), maybe this is something we’ll see coming to Lightroom Mobile at some point?

[via PetaPixel]