Real-life CSI: Scientists decode 3D scenes from reflections in human eyes

I like CSI TV shows (even the cringeworthy ones). While many of their methods of solving cases are pure fiction, some are actually becoming reality. A group of scientists from the University of Maryland has developed a method that reconstructs a 3D image using reflections found in human eyes. How cool is that?

[Related reading: Stalker attacks pop star at home; found her by studying selfies catchlights]

I know you’ve seen, and maybe taken, countless photos of people, and even just their eyes up close. In those eyes, we can see the reflections of the subject’s surroundings. For studio portraits, these reflections also tell us about the lighting setup that was used.

How did they do it?

To create a 3D scene from these eye reflections, the researchers faced two big challenges. First, they needed to guess the eye’s pose accurately. Second, they had to untangle the eye’s iris texture from the reflected scene. By fine-tuning the cornea poses, depicting the scene’s radiance field, and refining the observer’s iris texture, they cracked the code. The team also introduced a simple technique to enhance the iris texture pattern, which further improved the quality of the 3D reconstruction.

The team explains that the cornea geometry is remarkably similar in all healthy adults. This consistency allowed them to calculate the position of a person’s eyes in the image by measuring the pixel size of their cornea. With this data, they trained the radiance field – a 3D scene made of light rays – on the eye reflections. These rays, shot from the camera, bounced off the approximated eye geometry.

The tricky part, though, was separating the iris texture from the reflection in the eye. They can intertwine and give inaccurate results when creating a scene reconstruction. So, the team made a 2D texture map that learns the iris texture, effectively removing the iris from the final 3D scene. Determining the eye pose from just the image was a bit noisy. To deal with this, the scientists optimized the eye pose, significantly improving the performance of the reconstruction.

The team put this technique to the test using music videos from Miley Cyrus and Lady Gaga. They managed to reconstruct an object appearing in Miley’s eyes and identified an upper body-like object in Lady Gaga’s eyes. Though the quality of these videos made it hard to verify the correctness of the 3D reconstructions, these early experiments show that this method has the potential to revolutionize how we perceive and capture the world around us.

Potential uses and misuses

This breakthrough is more than just a fun trick, of course. It has the potential to unlock countless new possibilities in photography, surveillance, and maybe even forensic science. Definitely not yet, but the potential is there and we may see the technique from cringeworthy CSI shows in real life.

But just like everything else, this potentially useful technology has a flipside, too. It made me remember the time when a stalker found a pop star in Japan by studying the reflections in her eyes. Tools like this open can lead us to keep even less of our privacy and make us vulnerable to all kinds of, ahem, unstable people. Sadly, like every new piece of technology – it’s not about the tool itself, but about the person who uses it. Still, I hope that this will develop further and make a breakthrough in crime solving, but also in art and photography. I’ll try not to worry about the dangers… For now.

[via Gizmodo; video credits: World-From-Eyes.github.io/]