Windows have been ruining photos ever since the first time a photographer tried shooting through one.
Unless you bring along dedicated contraptions or start messing around with cloths and funny angles, shooting through the glass will likely lead to an annoying reflection that will make you want to smash it to pieces. (If you’re actually trying to get a reflection then scratch everything I said; windows are awesome).
This problem might soon come to an end, though; as researchers say they’ve developed an algorithm that can automatically remove reflections from digital photos. The algorithm can’t remove all types of reflections, but it does an impressive job with the ones it can remove.
In order for the algorithm to work the photo must include a set of almost identical reflections, says YiChang Shih of MIT.
The first author on the paper, who just completed his PhD in computer science, say double-paned windows create the perfect reflections for the algorithm. “With that kind of window, there’s one reflection coming from the inner pane and another reflection from the outer pane”, he explained. ”But thick windows are usually enough to produce a double reflection, too. The inner side will give a reflection, and the outer side will give a reflection as well”.
Shih’s co-authors include his thesis advisors – professors of computer science and engineering Frédo Durand and Bill Freeman, and Dilip Krishnan – a former postdoc at MIT and current Google Research employee.
Once the group had the desired double slightly offset reflection, they discovered that the standard techniques used in camera vision wouldn’t work for their purposes.
Eventually they used a new technique co-developed by another of Freeman’s postdocs, Daniel Zoran, and another researcher named Yair Weiss.
MIT News explains how this technique works and helps the final goal of removing reflections:
“Zoran and Yair Weiss of the Hebrew University of Jerusalem created an algorithm that divides images into 8-by-8 blocks of pixels; for each block, it calculates the correlation between each pixel and each of the others. The aggregate statistics for all the 8-by-8 blocks in 50,000 training images proved a reliable way to distinguish reflections from images shot through glass”.
Yoav Schechner, a professor of electrical engineering at Israel’s Technion, said that there have been previous attempts to eliminate reflections, but they had only partial success. One of the drawbacks of these methods was that they were unable to automatically distinguish between the desired image and the reflection.
MIT researchers have created a new algorithm that, in a broad range of cases, can automatically remove reflections from digital photos. On the left is the original photo taken through a window, with the photographer’s reflection clearly visible. On the right, the reflection has been separated from the photo. Credit: Courtesy of the researchers
“The ideas here can progress into routine photography, if the algorithm is further robustified and becomes part of toolboxes used in digital photography,” Schechner adds. “It may help robot vision in the presence of confusing glass reflection”.
