The death of CAPTCHA image tests – Bots find traffic lights faster than humans

Aug 10, 2023

John Aldred

John Aldred is a photographer with over 20 years of experience in the portrait and commercial worlds. He is based in Scotland and has been an early adopter – and occasional beta tester – of almost every digital imaging technology in that time. As well as his creative visual work, John uses 3D printing, electronics and programming to create his own photography and filmmaking tools and consults for a number of brands across the industry.

The death of CAPTCHA image tests – Bots find traffic lights faster than humans

Aug 10, 2023

John Aldred

John Aldred is a photographer with over 20 years of experience in the portrait and commercial worlds. He is based in Scotland and has been an early adopter – and occasional beta tester – of almost every digital imaging technology in that time. As well as his creative visual work, John uses 3D printing, electronics and programming to create his own photography and filmmaking tools and consults for a number of brands across the industry.

Join the Discussion

Share on:

It’s quite ironic that a system designed to be bots is being beaten by bots faster than humans can get through it. But that appears to be exactly what’s happened with CAPTCHA. You know, all those tests you have to do to prove you’re not a robot on a website?

Well, robots are better at proving it than we are now, according to a new study. The study put its AI against a thousand real humans on different CAPTCHA systems on over a hundred popular websites. We lost.

What’s CAPTCHA?

CAPTCHA, for those who didn’t know, stands for “Completely Automated Public Turing test to tell Computers and Humans Apart”. It’s a pretty self-explanatory title. It’s a test to see if the person interacting with the website is human or not.

If you’re reading this, you’ve almost certainly experienced a CAPTCHA on a form somewhere during your time on the web. They exist pretty much everywhere. Even Google pops one up occasionally if it thinks you’re searching for too many things too quickly.

Sometimes, you just have to check a box that says you’re not a robot. Sometimes, you might need to decipher a code, and at other times you might be asked to identify all of the squares containing traffic lights. Or all of the photographs with a pedestrian crossing. Yeah, those. The really annoying ones you end up having to do about fifteen times before it finally lets you in.

However, with the advancements in AI over the last little while, bots are now better at beating these tests. And they’re better at beating them than we are. They’re beating us on both accuracy and overall speed – sometimes by a massive margin.

The researchers tested their system on over 100 popular websites. They also tested a thousand humans of varying ages, genders, education and locations on the same websites. While the humans, as a group, appear to cover the same range of websites, each individual human reviewed no more than ten websites.

But, why?

The point of the tests, was to essentially show that the more than twenty-year-old technology is no longer enough to keep your websites free from the bots they’re designed to protect you from. The researchers are calling for better methods to determine which users on a website are actually human.

Bots have been causing havoc on the web for as long as the web has been a thing. Back in the day, most of them were friendly and helpful. Search engine crawlers and the like. These days, there are probably more malicious ones than benign ones. And wherever possible, you want to keep them out.

Hence, the CAPTCHA tests.

The tests are designed specifically to be easy to solve by humans but difficult for a bot to solve. Most bots up until lately have been pretty dumb only doing what they’re told and they’re often coded in such a way that the first time they see any data they don’t understand, they just give up.

So, the tests beat those easily. Things have changed now, in these latest tests.

So, how did we do?

First up, accuracy. The humans in the tests beat the CAPTCHA 50-84% of the time, depending on the specific test. The AI bots, on the other hand, beat the tests 85-100% of the time across their range.

When it comes to text-based CAPTCHA tests, humans beat the test in 9-15 seconds. The accuracy for humans isn’t specified, but it’s largely irrelevant. At least, it will be when you see what the AI got. In the same tests, the bots got a 99.8% success rate with a response time of under one second.

The bots’ accuracy ranges from 85-100 per cent, with the majority above 96 per cent. This substantially exceeds the human accuracy range we observed (50-85 per cent)

On the other hand, the CAPTCHA photo tests appear to be a bit of a different story. The AI isn’t terrible. Well, I suppose, depending on your perspective, it might be. But the short version is that humans and bots are pretty much neck-and-neck when it comes to image-based CAPTCHA systems, at least for the moment.

This is very likely to change soon, given the rate at which other AI technologies are advancing. But even if they’re on level pegging with a human, the system as it stands right now is still broken.

What’s also potentially likely is that the web’s most annoying photographs are soon going to disappear forever!

[via The Independent]

Filed Under:

Tagged With:

Find this interesting? Share it with your friends!

John Aldred

John Aldred

John Aldred is a photographer with over 20 years of experience in the portrait and commercial worlds. He is based in Scotland and has been an early adopter – and occasional beta tester – of almost every digital imaging technology in that time. As well as his creative visual work, John uses 3D printing, electronics and programming to create his own photography and filmmaking tools and consults for a number of brands across the industry.

Join the Discussion

DIYP Comment Policy
Be nice, be on-topic, no personal information or flames.

Leave a Reply

Your email address will not be published. Required fields are marked *

5 responses to “The death of CAPTCHA image tests – Bots find traffic lights faster than humans”

  1. Jeff Grubbs Avatar
    Jeff Grubbs

    Is it a surprise? The whole purpose of CAPTCHA lately has been to train AI what is a traffic light, bicycle, train, etc.

  2. Stephen Jenner Avatar
    Stephen Jenner

    I guess that what humans are good at is knowing when their ‘bore’dom threshold has been reached, and they move on.
    If too many of those capture thingies precede my ability to view or read a website, I don’t bother with it. It was probably not worth considering anyway.
    If those bits that I missed are any good, they will show up somewhere else, sometime later.

  3. Just/INCREDIBLE Photography & Digital Artistic Illustrator Avatar
    Just/INCREDIBLE Photography & Digital Artistic Illustrator

    I beg to differ. I can do it faster than AI.

    1. Ingahy Merina Avatar
      Ingahy Merina

      Just/INCREDIBLE Photography & Digital Artistic Illustrator too late AI already moved on something else. You are too slow

  4. Luis von Ahn Avatar
    Luis von Ahn

    The OCD nerd in me must mention that you probably wanted to say “a system designed to BEAT bots is being beaten by bots”. But the irony is probably even a level deeper: After (ab)using humans to solve CAPTCHAs for crowdsourcing OCR, to decipher house numbers, gather geographic information and improve AIs’ image segmentation and recognition capabilities…

    https://smartermsp.com/pioneers-in-tech-luis-von-ahn-turns-everyone-into-a-crowdsourcer/
    https://blog.goodaudience.com/how-we-all-helped-unknowingly-google-to-digitize-books-acb45bc65084
    https://d3.harvard.edu/platform-digit/submission/recaptcha-the-brilliant-business-model-that-only-one-man-could-create/
    https://www.sciencedirect.com/science/article/abs/pii/S0303243415000665

    … we are surprised that tech actually became good at it. Perhaps, humans already are lightbulbs in a Matrix sense of the word.