Facebook has relatively recently introduced the so-called “reactions” to posts. But soon, instead of clicking on the heart or a laughing emoji, you will be able to “react” with your profile photo. The researchers at Tel Aviv University and Facebook have come up with a method to bring your selfies to life. All they need to do it is a single photo, and the resulting animation is pretty impressive. It seems like your photo is actually a short video, and it’s incredibly accurate considering that they only use one 2D image for the animation.
According to the study, the team uses a “driving video” of an “actor” to map out facial expressions of various emotions. You can see the examples in the video how they use a “model face” to act out being angry, happy, surprised, sad and idle.
The team performs mapping of the facial features in both the driving video and the image. Other than mapping the regions on the face, they also add reference points outside the facial region to make the animation more realistic. But it doesn’t end here. The facial expression is transferred from the video onto the 2D image, but it’s still not entirely realistic. So, the researchers have worked out the method to add the fine-scale details we associate with facial expressions. As a result, the animated picture gets tiny wrinkles and the recreated hidden regions, like those inside the mouth. You can see it in the example of the “angry” animation:
According to the researchers, the biggest challenge was to keep the identity of the target face while applying the expressions from the arbitrary driving video. If you ask me, I think they did a really good job.
Other than applying it to portrait photos (selfies), this method can also be used on photo-unrealistic pictures, like paintings and even emojis. It looks fun (and slightly creepy) to see a painting of Frida Kahlo with the animated crooked smile:
The researchers mention that photos animated like this “resemble the moving portraits from Harry Potter’s magical world, where people in photographs move, wave, etc.” The team writes that their primary application is “creating reactive profiles,” or avatars that “can react to different stimuli.” So, I guess this will be particularly interesting for the Harry Potter fans. I’m not among them, though, but I still find this pretty amazing. I hope they’ll add the eye-rolling feature, too. I would use that one on Facebook a lot.
