Ask HN: Do we need Google captcha? | Hacker News Reader

Ask HN: Do we need Google captcha?

71 pointsfractalf79 comments

Many of us remember the early batches of captchas. Annoying and sometimes next to impossible to get right. Googles "one click" captcha was a fresh breeze when it came. I'm not so sure anymore though. Now instead we have to use brain power to press 2-4 images in an AI quiz or said differently "help Google improve their ML data-set for free" which I'm not at all comfortable with. Isn't there a better way to fight bots? Could not the community make a FOSS alternative? How hard is this really?

View on HN

Two weeks ago my website [0] got hit by a botnet attack. These bots were submitting 2 POST requests to /login, followed by 2 POST requests to /signup, and since there were no checks in place, they were getting in quite easily. All their requests originated from residential IP addresses (probably hacked IoT devices) so IP blocking seemed impossible. I knew I needed to do something, but didn't want to send my users' data to Google.

I settled on implementing a honeypot technique using invisible_captcha gem [1]. It uses multiple techniques to detect bot activity, such as checking if the form was submitted too fast, hidden form attribute only visible by bots but not humans, etc...

Luckily for me, these bots weren't quite sophisticated so I managed to clamp down on their activity. I still get a couple of bogus signup attempts every day.

Disclosure: I am a contributor to the invisible_captcha gem

[0] https://tryhexadecimal.com

[1] https://github.com/markets/invisible_captcha

EDIT: grammar and wording

Very interesting idea (the invisible captcha approach)! Since you are a contributer, can you tell us how efficient this strategy is? I mean, do you still experience advanced bots being able to still act as humans?

They still try to sign up, but none of them can actually bypass the captcha. I implemented a custom callback that returns a HEAD response with 404 to fool them (because I noticed that attacks intensified when I returned 2xx or 3xx responses).

THIS is what I'm talking about :) Great work! Seems to me this is very centered on Rails apps. Would it be possible to do somethink like this for "everyone"? I have nothing agains Rails, but as a js/php/python develper myself, I wouldn't know where to begin with this..

The easiest way to start would be introducing an invisible form field [0] and on submission, checking if it is not empty. Give this form field a random name so that it wouldn't be populated by password managers.

You can take it one step further and check how fast the form was submitted. If it is below the predefined threshold, it is probably a bot.

[0] To hide the form field, you can use one of these snippets, interchangeably:

    "display:none;"
    "position:absolute!important;top:-9999px;left:-9999px;"
    "position:absolute!important;height:1px;width:1px;overflow:hidden;"

It should be stated that you should either use the first approach (display none) or give a 0 height/width, so that screen readers don't pick up these fake fields and ruin accessibility on your form.

Unfortunately anything screen readers can recognize as invisible, a more sophisticated bot can too.

There's also aria-hidden to prevent screen readers from touching the honeypot field.

It's worse than that. It fingerprints the browser, the OS, and a ton of other stuff.

I'd love to see a solution that's privacy-friendly.

not possible because anything based off computation only wards off denial of service attacks, not bots which usually have too much profit.

the absolute best you could do is track on the site itself without google, but you still need to track user's data.

anything else you can think of can be beaten pretty easily or is going to really hurt your user experience to the point of severe loss of revenue

any CAPTCHA custom to a site will be broken very easily if your business takes off.

> any CAPTCHA custom to a site will be broken very easily if your business takes off.

For me, this is an interesting and open question. I've taken the view that as long as it's easier to come up with new types of challenges than it is to create an automated means to solve those challenges, CAPTCHAs offer some protection. If you're ok with requiring JavaScript, mouse input, and images (e.g. you don't require your site to be accessible by sight-impaired users who cannot use a mouse), I think people can come up with varied enough challenges to present a problem to bots. How long will that be the case? I don't know.

The problem is coming up with challenges that are not trivial to automate and not annoyingly hard for users.

Botters aren't just making passive income, they can fix their bot every day as long as it's making them money. Many sites try "random" questions which are really easy to pattern match with a little regex.

The amount of effort that goes into botting generally scales linearly with userbase size (not always). But once you're talking about a botter with a profit motive when your site has hundreds of thousands or millions of MAU, I think trying to come up with challenges to beat botters would be prohibitively time consuming.

What if you had a community helping out, somehow gamify making new challenges? I dunno, just have the feeling if there was a proper project with enough bright minds behind it, it should work

This sounds like a recipe for disaster when 4chan and friends find out about it.

> The problem is coming up with challenges that are not trivial to automate and not annoyingly hard for users.

> I think trying to come up with challenges to beat botters would be prohibitively time consuming.

I agree that is challenging, and I think this question will need to be settled empirically. Sub-questions include:

a) What is the threshold for bot operators to engineer custom CAPTCHA solutions? (e.g. for a large website, how much can a bot network make?)

b) Even if it were not possible to conclusively beat the bots, what is the threshold at which bot operators decide that they would rather make their money off of other sites instead?

The way you should think about captcha is how much does it raise the cost to bot. You can always just pay people to click to bypass

In other words, Google needs captcha and the data it provides.

I work on hCaptcha -- we pay websites for providing data labeling via our captcha. We also provide a mechanism to accept privacypass as a bypass.

There are three major categories of bots: 1) a combinatoric attack hoping to get lucky with an answer 2) employ some amount of smarts to fool the captcha (fetching lots of images without answering anything until they are sure of the answer, replay attacks to mimic human behavior, perceptual hashing of images etc) 3) Anti-Captcha (using people to solve captchas).

We can mostly recognize the first two kinds of attacks, but it is very difficult to stop the anti-captcha humans -- especially without tracking as many features as possible. One of the chief issues with most captchas is that they do not provide epidemic analysis and a webhook/endpoint to inform sites about a user after the initial sign-up. This also requires some amount of user tracking to recognize malicious behavior and retroactively tag and act upon it. Monetizing human activity can somewhat offset costs associated with dealing with these networks of bad accounts later and using privacypass gives the user a mechanism to bypass tracking their activity while also pricing out malicious account signup using anti-captcha.

A privacy-respecting captcha used for monetization also opens up an entirely different opportunity for publishers, offering a lower friction experience for users who want to support online content without privacy invasive advertisements and paywalls.

Very interesting idea, thanks for sharing! Is this solution doing any form of tracking or is it purely just sending solutions to your puzzles?

We can't really track you across the web in the same way that Google can, but we do track your activity on a particular website and how you interact with the captcha to see if it conforms to the model of a human user of that website.

In the case of Google tracking you, they have your personal information and they are using your activity to show you ads -- we don't have either of those aspects to our business model; so our tracking is essentially anonymous.

My problem with Google Captcha is that I have to solve the "select all" window every time I check the checkbox, even when I'm logged in to Google. Maybe it's because I use Safari, but it's getting really annoying. What's the benefit of Google Captcha if you have to solve the captcha every time?

That's what they show if you're not on chrome, especially if you're blocking scripts or on a VPN.

I use Firefox all the time and don't have to solve the image problems, however I always have Google cookies and Javascript enabled.

In my experience it doesn't work at all if you're blocking scripts.

It does not work under Chromium either with cookies disabled either.

From a user point of view on Firefox, Google captcha is a an absolute hell.

Every single time I have to pass few batches of captchas. I'm not on a private window and I'm logged in to Google. So I started to blacklist websites that use Google captcha. and forced myself to change reading habits.

Exactly what I was thinking when writing this question. Seems to me there really is a big demand for something proper, secure and open source. Kind of like how letsencrypt help secure the net

There were a few factors making letsencrypt successful: alternatives were not free, and mainstream browsers were all making attempts to surface unsecured http clearly on address bars.

A free open source alternative for captcha will have a harder time getting traction, mainly since the popular option (Google) is already free; and neither business not customers (who are not privacy sensitive) care about the end result unlike the browser and http case.

I have been prompt by google captcha many times here by the HN login.

We turn that on when HN is under certain kinds of attack: https://hn.algolia.com/?dateRange=all&page=0&prefix=true&que...

If you're experiencing it over a long period of time, it might be good to email hn@ycombinator.com. I don't think that should be happening.

CAPTCHA isn't just a matter of protecting your site. One of the most evil attacks nowadays is "Distributed Spam Distraction", where you spam your victim with thousands of emails per second so an important email (e.g., fraudulent purchases) gets lost in the noise.

How do you do this in a world with decent spam filters? By using the victim's email to sign up for real services so they get hit with a welcome email. Because these are real services, spam filter won't catch it. This can only be done with services that have sign up forms that are easily automated.

The most evil thing here is your email is crippled even after the attack is over because these real companies will keep sending you newsletter and it's impossible to unsubscribe to them all.

You've just reminded me I really need to use unique email addresses for each service.

If you use gmail, you can add a + followed by anything and it goes to the same mailbox.

For example, if signing up to drop, I might use myemail+drop@gmail.com

Makes it very easy to see which services are selling the address you provide to advertisers

Yeah I know, and you're unfortunately correct that I use Gmail, but it's something I'm planning to change soon.

Also, if someone was targeting you with spam that won't help. They'll just remove the "+..." and you're back to the same problem.

I didn't even find Google Captcha to be effective, I still got a ton of spam coming through my web contact forms - often advertising software tools to bypass Captcha.

In the end I wrote my own rule based spam filters and that has been significantly more effective without tracking or annoying users.

Over the past 2 years or so I've become incredibly skilled at identifying crosswalks, fire hydrants, traffic lights, and bicycles.

I don't necessarily mind contributing to googles car-driving AI, but the services that I've put this captcha infront of have seen a dramatic increase in spam.

I unfortunately don't know what the solution to this problem is but I would gladly contribute to whatever someone smarter than me comes up with!

>identifying crosswalks, fire hydrants, traffic lights, and bicycles. I don't necessarily mind contributing to googles car-driving AI

Wow. I just realized that's exactly what they must be doing with the data!

I believe what the spammers have started doing is presenting these captchas to human users on more illegitimate sights, such as those to watch free sports broadcasts, etc to gain access.

The user solves the captcha for the site which then uses that solution to fool the actual Google captcha.

I only know about Buster:

> The difficulty of captchas can be so out of balance, that sometimes they seem friendlier to bots than they are to humans. The goal of this project is to improve our experience with captchas, by giving us easy access to solutions already utilized by automated systems.

https://github.com/dessant/buster/blob/master/README.md

Probably not.

A sophisticated attacker will simply use something like Mechanical Turk to get past Google captcha, and Google captcha is incredibly aggravating to a number of marginal users. But it's simple to implement, so that makes it popular.

I'd even say that most services don't need captcha in the first place. Captcha only affects relatively unsophisticated attacks, and if it's just an entry in your database, who cares? If it's sending you spam in an attempt to DDOS your email or customer service platform, use correlation between purchases and accounts to filter out the spam.

So, as with all security related matters, identify your threat model, and use that to figure out what you actually need, not what you think you need.

Are bots now advanced enough that transcribing obscured text is no longer a viable solution?

There's dozens of simplistic captcha libraries that serve obscured text. I can would rather transcribe a few letters much than complete a Google Captcha, probably faster, too.

They were advanced enough to do that 10 plus years ago.

Here was an earlier discussion about it:

https://news.ycombinator.com/item?id=20158386

My general answer is "no". But the question also isn't meaningful unless you define what it is you're trying to protect.

(E.g. maybe it's reasonable to have a captcha of some sort for account creation, with fallback to a human-involved process. But if you're trying to prevent credential-stuffing on a login form, the correct approach is using the Pwned Passwords dataset and/or cooloffs.)

To the extent that you'd consider non-Google alternatives, there are multiple different CAPTCHA services, and we at NetToolKit recently launched our own version called Shibboleth: https://www.nettoolkit.com/shibboleth/about

You can try out demos at https://www.nettoolkit.com/shibboleth/demo

While not open-source, the service is very affordable ($10 for 100,000 CAPTCHAs), and clients get to review user submissions to see if the CAPTCHAs are reasonable or not.

Would love to get your feedback on the service.

The maze one is interesting, but it seems it can be frustrating (because it didn't enforce walls when I tried, though I can see why it might do that)

Yes, the concern is that since browsers interpret the end-user code, if it has knowledge of the walls (required to enforce them), bot networks might be able to use the JavaScript code to figure out the correct solution in an automated fashion.

This maze captcha looks trivial to defeat

I understand why it might seem that way. I personally don't think it would be trivial to wire up a working solution (including generating the correct solution strings); if you do, I certainly would welcome you to post your solution.

I estimate maybe 3 hours of effort. Will you pay me for my time upon completion ($100/hr)? If so I'll do it

if 3 hours is too much I'll crack the naval command captcha for 1 hour of work.

Adversarial testing -- I like it. Contact me via email (see my profile) and let's see if we can agree on the terms (again, it's more than just solving the puzzle). We would want to set up a proper test bed, and I'll be travelling next week, so it might take a couple of weeks to get everything ready.

I understand why you'd call it that but won't it be confused for Shibboleth, the single sign-on log-in system?

https://en.wikipedia.org/wiki/Shibboleth_(Shibboleth_Consort...

Maybe. I'm hoping that single sign-on and CAPTCHA are sufficiently different that people won't get confused. Also, the dev team behind this didn't want a generic name, and there are enough products/services out there that name collisions are pretty easy to encounter, unless wholly making up a name (which might then require more explanation).

Just tried it out.

Clicking on the buttons with a laptop trackpad was tedious and unintuitive.

It would be nice if I could use the arrow keys (or even WASD) on my keyboard.

Other than that it seems preferable to Google's new image captchas.

Thanks for the feedback! One thing that is not clear from the demo (and something we should fix) is that you can click on the path CAPTCHA to grant it focus. After that, you can use the arrow keys (although WASD currently does not work).

Ah! I tried clicking every except on the path itself!

> WASD

Please don't, not everyone is american, not everyone uses a qwerty keyboard.

You should add arrow key mapping, so we can beat the captcha with the keyboard. Using the mouse is tiedous.

Yes, thanks for that feedback. We actually do have support for arrow keys, but you have to first click the path to focus on it (made similar comment at: https://news.ycombinator.com/item?id=21327738)

I started using Buster[1] few months back with Google Speech to Text API. I use Firefox on Linux and I hadn't had to manually fill a captcha in all these months. Highly recommended. :)

I get 60 free minutes/month from Google Speech to text API. So, It's pretty much free to use.

[1]: https://addons.mozilla.org/en-US/firefox/addon/buster-captch...

I like captchas that ask a question relevant to the topic of the site. For example, a forum about D&D might ask: "Who created D&D?" and accept all answers that contain "gygax", "arneson", "TSR" or "WotC". Trivial for your target demographic, but at least it requires a very different kind of bot.

Though I suppose if this becomes common, bots might just google that question and try some of the zero-click results as answers.

Honest question. What’s wrong with Google Recaptcha V3?

No need to select 25 traffic lights. Gives you a score from 0 to 1 so you can decide whether you let them in easily or not.

If Google doesn't like a client (e.g. Firefox users) they can just set their score to 0 and break half the internet.

You could say that for any captcha library though.

But the risk is doubly so for a closed-source codebase maintained by a company with a history of doing this sort of stuff.

It's impossible for me to pass Google Recaptcha V3, so sites that use it are sites that I cannot use.

Have you ever been locked out of the internet because Google thought you were a bot? It has happened to me before, and it's not fun. Your only recourse for sites running v3 is to install Chrome and sign into a Google account.

You have to include the script on every page of your site, not just the page with the form.*

* https://news.ycombinator.com/item?id=21324523

reCAPTCHA v3 will actively block many genuine users with no recourse, in a way that is incompatible with various accessibility and disability legislation. In the US I believe it’d be illegal for any businesses to gate functionality on reCAPTCHA v3 failure alone, and in AU it would be for a government site but I’m not certain for businesses.

reCAPTCHA v2 kinda had this problem, but Google could scrape by because it wasn’t obvious that some users were being flat-out blocked, though it does seem to be the case.

In reCAPTCHA v3, Google get around that problem by putting the burden of supporting some alternative for such denied users onto you, rather than on them—so they recommend things like falling back to sending an SMS code to validate such users. (Given how they position the product, this is basically just a disclaimer of liability—“we told you to do something extra! (though not very hard)”. I have no idea how legally effective that approach is, but that’s what they’re going for.)

That’s how we do at Fastmail for signup (which is a substantial fraud target): we use reCAPTCHA v3 if we can (and only on the signup page, I may add), but if it won’t load or if it flunks the user, then we fall back to requiring SMS verification, so that real people should still be able to sign up.

So that’s the major problem with reCAPTCHA v3: you can’t, or at least shouldn’t, use it by itself.

Add to the problem the complication that you need a replacement for the captcha that is usable by users with disabilities.

Hcaptcha will soon be open source.

> Isn't there a better way to fight bots?

Don't?

I mean, what are you trying to protect? If it's account registrations, shouldn't an email confirmation be enough?

why would an email confirmation be enough? playing whack-a-mole all day with different domains is unreasonable. whitelist only of domains is awful, and botters have tons of popular email accounts.

> If it's account registrations, shouldn't an email confirmation be enough?

This won't stop someone with enough sophistication.

Not every service will be targeted by sophisticated bots. What kind of service are we talking about? Is it even social? Is it even a service or just a public website?

OP did not provide a threat model or any description of the service really.

Talking about this "in general" is awful and leads to stupid bullshit like endless CloudFlare captchas on literally STATIC blogs.

E-mail confirmations are easy to circumvent.

Plus if your app will send email to any address someone submits without even rate limitations, your domain reputation can take a hit for sending to fake addresses.

Familiar with email aliases (ie user+words@gmail.com)? Easy to have infinite addresses and yet a single mailbox with automations upon receiving the confirmations.

Reply on news.ycombinator.com