CAPTCHAs can still detect AI agents

I actually saw a pretty decent captcha the other day on a Chinese website (I think Taobao? I forget.) anyway the cool thing they did was that the text wasn’t in an image it was a looping video, but the text in any one frame was incomplete (only parts of the Chinese characters). And each frame different parts of the characters were visible, with a lot of noise in other parts of the frame where parts of characters would have been in other frames. A human brain sort of smoothed this out between frames and sees the characters clearly, but taking a screenshot was impossible. And becuase I don’t know Chinese I wasn’t able to take a screenshot and ask AI to translate the message. It seemed like a pretty good anti AI method. Of course an algorithm could be made to convert the video into a single frame, but captchas have always been defeatable by a sufficiently motivated attacker, they are only to raise the bar slightly against the swarm of dumb bots.

    > AI does not complete CAPTCHAs like humans. If you look across all the data of humans and AI completing CAPTCHAs, you start noticing differences in features like error patterns. Our recent paper found statistically significant differences across sequential click patterns, direction changes, and overselection behavior - features that define how a participant, agent or human, would solve the CAPTCHA problem

putting aside the possibility that if bot makers wanted to they could work on these problems, if you need to perform statistical analysis in a captcha setting you have already failed. bots don't stick to a given session persistently so there is no useful profile to form. at best you may improve on IP reputation scores (and they probably already do) but that doesn't help much.

But.. the task was never "detect this" but always "detect this within acceptable constraints".

Sure, once you collect enough bits, you can tell that its me. And if you know from other sources that I am human, that solves your immediate problem. But if you do that, you have still failed at the task of detecting certain kind of abusive behavior without harming my anonymity.

This feels like the kind of thing where, "you must be at least this human to pass" and that it just otherwise mostly wastes your time if you're a robot would cover most of what Captchas are useful for.

Like, if it takes you 3-5 seconds to get through a captcha as a human, as long as every single event has that effort added, the impact to something trying to use/reuse the end-page is way worse if you're a robot than if you're a human.

I can see a few usecases where it would still be valuable to continue the game of cat-and-mouse, but I feel like solving for consistency of human experience of your website, may actually be more punishing to anything trying to bypass it.

Thanks all for the discussion! Would like to highlight two parts that maybe didn't come fully through, and we'll work on making this clearer:

1. CAPTCHAs can still detect AI agents...if you know where to look. Most commercial CAPTCHAs are not doing the cognitive process tracing you see in our paper. Nor are they really doing 'behavioral biometrics' (but that is slightly tangential here). Our CAPTCHA example here is about repurposing the current paradigm with a new methodology (cognitive process tracing) in a way that is able to combat human/machine discrimination in a way that's independent on frontier AI progress.

2. There are lots of concerns about adversarial robustness, which are very fair, and we reported some fine-tuning tests in the paper. Generally, there are two mental models for me that work, both framing fraud as an economic game.

First, compare AI spoofability concerns to something like a passport or a fingerprint. The cost to mimic continuous cognitive and behavioral patterns over time seems more computationally complex. In other words, sure this method is not bulletproof with infinite resources, but nothing is. We rely on defeasible mechanisms everyday, and our job is to make that significantly securer.

Along these lines, there's a common line of criticism that suggests once fraudsters know the game, they will solve the game. The CAPTCHA presence in the 2000s didn't mobilize massive deep learning / image recognition advances from the fraud community. Nor are these same bot farms solving quantum computing despite there being immense incentives to. If anything, the real threats are stuff like JavaScript injections, not really fully simulating human cognition

Apparently CloudFlare’s turnstile can’t, as evidenced by several public-facing CRUD and mail routines we maintain that no longer are warding off the spam.

CAPTCHAs are great. Exploiters get around them with proprietary anti-detect browsers and unethical residential proxies, while privacy browsers and affordable privacy VPNs get blocked and shadowbanned to death.

Fingerprint.com, while not a CAPTCHA, gives you +3 suspicious score just for using privacy settings like adblock on your browser. This makes it harder to sign up for any sites that use fingerprint.com.

https://github.com/CloakHQ/CloakBrowser is a good anti-detect browser as well as CAPTCHA bypass which is honestly fun to use coming from privacy browsers because every site just works and captchas get solved.

I wonder if AI could be detected via copyright. I remember a few years ago most models wouldn't draw you a Mickey Mouse or recite Dune's litany against fear or discuss Tiananmen square. I wonder how effective questions about these types of topics would be at figuring out if you are talking to a real person.

As a crude joke that is only tangentially related, I saw a skit video a while ago with two guys saying goodbye and one says "send me a dick pic when you get home" and then explains that an AI won't simulate it so this is a sure way to know that it's his friend confirming his safe arrival.

I had to do a Captcha the other day, and the letters looked awful, so I clicked the speaker for an audible Captcha instead. I was even more horrified. The sound was almost painful. Sharp noise blasting as a high pitched tinny voice bellowed numbers at me. I honestly don't know how blind people use the internet these days with such blockers in place, and that's kind of sad. The cookie banners, the captchas and the bots and laws that made both appear have kinda en$hittified humanity's greatest communication tool.

What happened to adversarial attacks? I.e. noise that makes an image look like something else to a classifier than to humans. I guess frontier LLMs are no longer vulnerable to those?

I’ve been using Claude Opus 4.7 with Chrome MCP, and it has worked successfully about 95% of the time. However, I’ve failed various hCaptcha challenges.

I think it's just a game of cat and mouse. It might be easier to catch naive AI agents that are not fine-tuned for specific CAPTCHA tasks with human behavior, can't recognize new challenges, don't know when to stop and ask a human, and just want to brute force their way with limited or no specialized harness and tools available.

Appreciate this article...shows some interesting insights on how humans "behave" vs agents.

So now I have to fail the capcha to prove I'm human, but in the right way? We don't hate these people enough.

- LLMs can't learn, therefore, LLMs are only good for things on which they are trained. - Captchas are not friendly with trial and error, so agentic solutions also don't help. - It's impractical to train LLMs on everything. - We humans are capable of creating infinite ways of captchas.

While each of these sentences is true, captchas will always win against LLMs.

Just ask, "I need to wash my car. If a carwash is 50 ft away should I walk or drive?"

Until they learn to do that. So cat and mouse. So nothing new.

Captchas are primarily to punish users for not allowing tracking, or using the “right” services, they may prevent some bots as a side effect (or a pretence from the provider) but it’s mostly for google and cloudflare to abuse their monopolies.

yeah no. it is funny easy to make a mcp server and plug a qwen3.6 to it. it was more annoying to convince the llm that it can clear captchas than the actual passing

For real Bro!!!

Adversaries do not have to wait for LLM models to evolve to mimic human process, they can simply evade the detection JavaScript that evaluates similarity. JavaScript is visible, can easily be reverse-engineered.