Hacker News Reader

Comment by hombre_fatal | Hacker News Reader

Yeah, you can't really talk about downsides of Recaptcha/Cloudflare without also acknowledging the extreme amount of malicious actors and abuse on the internet.

We're in a "this is why we can't have nice things" predicament and you have malicious actors to thank for that, yet most people on HN only seem capable of attacking the few affordable solutions to that problem.

I'm even down with the theory that Cloudflare is a US government outfit, that's the only way I can wrap my head around such a generous free tier. But at what point does it worry you that the internet has so many fundamental issues that people willingly centralize behind such a large behemoth? How many options do I have when a kid is holding my forum hostage with a $5 booter service?

It's easy to shit on everything. Let's hear some real solutions.

View on HN

> Let's hear some real solutions.

It's by no means a full solution (there likely is no single full solution), and it may even be a bad solution -- but lately I've been trying to think about what the Internet would look like if we didn't have a massive arbitrage potential around server requests.

Part of the reason why everyone is trying to detect bots is because bots will very, very rapidly eat up your bandwidth and CPU time. We're used to offering our bandwidth/CPU for free to humans and either swallowing the cost if we're running a free service, or making up the cost in an adjacent way (ads, subscriptions, etc...). It's not bots that are the problem. It's that when someone asks our servers to do something, we do it for free. Bots are just a big category we can ban to make that problem smaller.

In many (but not all) cases, we shouldn't care about bots, and the only reason we do is because our systems aren't scalable to that level.

So I've been wondering lately what a server-defined per-pageload, or even per-request fee would look like on the Internet, maybe one that scaled as traffic got heavier or lighter and that was backed by a payment system that wasn't a complete rubbish privacy-disrespecting dumpster fire.

My immediate thought is, "well, everything would be expensive and inaccessible." But, the costs don't change. You still have to pay server costs today. Businesses today still need to make that money somehow. There are almost certainly downsides (all our current payment systems are horrible), but I wonder if it's more or less efficient overall to just be upfront about costs.

Imagine if I could put up a blog on a cloud service anywhere with scalable infrastructure. Then a post goes temporarily viral. Imagine if my server could detect it was under heavy load, detect that it was getting hit by bad actors, automatically increase the prices of requests by a fraction of a cent to compensate, and then automatically ask my provider to scale up my resources without costing me any extra money?

For a static site, suddenly I don't need to care if people or bots are hammering it, I don't need to care about anything except whether each visitor/bot is paying for the tiny amount of hosting costs they're hoisting on me. If bad actors start pushing traffic my way, I don't need to ban them. I just force them to pay for themselves.

> Part of the reason why everyone is trying to detect bots is because bots will very, very rapidly eat up your bandwidth and CPU time.

It is?

Thought bot detection was only done during registration etc. to stop them from sending spam etc. to real users.

If anything the javascript world we live in helps combat this. You need insane resources on the client just to have a page open. Several orders of magnitude more than the server need to generate and send that page.

In that case, an IP or IP block throttling is good enough.

Except then there are those pesky CGNATs to handle including Chinese Great Wall.

Anyway, high profile spammers will emulate enough of the browser to render any measure based on browser anomaly detection worthless. Including using a headless browser. The only way to defeat them would be too put some quite computationally intensive JS operation... (On par with mining, ruining all the laptops, phones and tablets. But you can make it not trigger every time.) This would make spamming expensive.

Server-side we have excellent AI spam filters that nobody seems to be using to fire off a captcha check later. The big problem here is that you cannot offload to some provider without inviting big privacy concerns. (Same problem as forum/chat/discussion platform providers.)

No. Botnets are large and broadly distributed enough to render protection methods based only on the IP or IP block ineffective. They're commonly used for mailbombing attacks such as those described here: https://www.wired.com/story/how-journalists-fought-back-agai...

Do you think a botnet with 10k machines is going to be meaningfully inhibited by making each machine's cpu run calculations for a second or two for each submission?

I'm sure reCAPTCHA looks at the IP and IP block as one of the inputs to its ML algorithm, but as one or two of perhaps a dozen different features - including mouse movement and/or keyboard input, which is quite a bit harder to fake.

high profile spammers will emulate enough of the browser to render any measure based on browser anomaly detection worthless

Based on actual experience of fighting spammers, that isn't the case. Like a lot of people new to spam fighting you're making assumptions about the adversaries that aren't valid.

There are many different types of spammers and attackers.

Some will be stopped by the simplest protection mechanisms.

Some will be indistinguishable from real humans, and you won’t be able to stop them without crippling your services for your real users.

But those are the two extremes. The real problem is the ones between those extremes.

Every intentional stumbling block you put in the path to try and stop those in the middle might also have a negative impact on your real users. The real problem is that the most troublesome attackers will learn and adapt to whatever stumbling blocks you put in the path. So, how many of your own toes are you willing to sacrifice with your foot guns in the name of stopping the attackers?

Very few, but that's OK. Good spam fighters don't have to sacrifice many or really any toes to stop nearly all spam. You seem to be assuming a linear relationship between effort and false positives, but that would be a very ineffective spam fighting team relative to the ones I've worked on. In practice you can have nearly no false positives combined with nearly no false negatives.

This isn't easy and many firms fail at it, but you it can be done and we routinely did it.

IP block for registration?

Seems highly unrealistic.

> automatically increase the prices of requests by a fraction of a cent to compensate

Great concept.

CPU, bandwidth, electricity, it's all just energy. And to a significant degree, money is just energy stored. I generate energy with my own work, store it in the form of money, and then transfer that energy to someone else, maybe to heat my home or cook me a meal.

Before money, I had to barter for those things. Maybe conceptually the internet is in a similar state at the moment. It doesn't have 'money'. Why can't I put CPUs in my wallet and then spend them? And why can't I charge visitors to my site by the CPUs they are costing me?

Instead, I have to, in a way, barter. For example, maybe I use ad revenue to earn my income, so I generate all this content, I barter that to the search engines, which barter with the advertisers, which barter with me, and I barter back to security guards to protect me from 'bad' actor bots. I'd really just like to receive CPU and bandwidth payments from them.

Isn't the reason we are freed from barter in daily life is because the government is intimately involved in the financial/banking system, and regulates it and issues money and so on? Maybe we continue to struggle with the internet because it started out unregulated and has never really transcended that because people insist on thinking freedom is best for commerce without appreciating the nuances.

There are alternatives to that. For all of the hype and vaporware of the cryptocurrency movement, the idea of digital-native programmable internet money is a powerful one. I’m personally excited by the idea of involving currency at the protocol level and having it interact naturally over tcp/ip and http. There is an alternative to ads if we can make it work.

> Before money, I had to barter for those things

Not at all. Barter was quite uncommon also unpractical. Most societies used (and use) social connections and trust.

I used to work on spam fighting.

This sort of solution is frequently proposed but doesn't work, because:

• Serving costs are rarely the problem. Normally it's annoying actions taken by spammers and the bad reaction of valuable users that matters, not the machine cost of serving them.

There are occasional exceptions. Web search engines ban bots because left unchecked they can consume vast CPU resources but never click ads. However, they only get so much bot traffic because of SEO scraping. Most sites don't have an equivalent problem.

• There is no payment system that can do what you want. All attempts at creating one have failed for various hard reasons.

• You would lose all your users. From a user's perspective I want to access free content. I don't want to make micropayments for it, I especially don't want surge pricing that appears unrelated to content. Sites that use more typical spam fighting techniques to fend off DDoS attacks or useless bot traffic can vend their content to human users for free, well enough that only Linux users doing weird stuff get excluded (hint: this is a tiny sliver of traffic, not even a percentage of traffic but more like an occasional nuisance).

• You would kill off search engine competition. Because you benefit from crawlers, you'd zero rate "good" web bots using some whitelist. Now to make a new search engine I have to pay vast sums in bot fees whilst my rich competitors pay nothing. This makes an already difficult task financially insurmountable.

The current approach of using lots of heuristics, JavaScript probes and various other undocumented/obscure tricks works well. Cases like this one are rare, caused by users doing weird stuff like committing protocol violations and such users can typically escalate and get attention from the right operators quickly. There are few reasons to create a vast new infrastructure.

That's how ads work. More visitors more pageviews/clicks. People who serve ads don't want to pay for bots which is why they are a problem.

Doesn't medium do this?

> That's how ads work. More visitors more pageviews/clicks.

That's not asking people to pay for bandwidth/compute power, it's selling something adjacent to your content that you hope makes up for the loss.

> People who serve ads don't want to pay for bots which is why they are a problem.

That's kind of my point. When you ignore the arbitrage potential of serving requests for free, it forces you to care about making sure that your content is only available to the "right" users. You have to care about things like scraping/bots, because you're not directly covering your server costs, you're swallowing your server costs and just hoping that ads make up the difference.

Theoretically, in a world where server costs were directly transferred to the people accumulating those costs, you wouldn't need to care about bots. In fact, in that world, you shouldn't care whether or not I'm using an automated browser, since digital resources aren't limited by physical constraints.

In most cases, the only practical limit to how many people can visit a website is the hardware/cost associated with running it. A website isn't like an iPhone where we can run out of physical units to sell. So if they're paying for the resources they use, who cares if bots make a substantial portion of your traffic?

> Doesn't medium do this?

No, Medium just sells subscriptions, you don't pay for server usage. As far as I know, no one does this -- probably in part because of problems I haven't thought of, also probably in part because there are no good micro-payment systems online (and arguably no really good payment systems at all).

The closest real-world example is probably AWS, where customers pay directly for the resources they use. But those costs aren't then directly passed onto the user.

If you had to pay for each request you would make fewer and limit the requests to serious requests (school, work, medical).

Having said that, you could provide a central service where people would buy credit to be used on many sites. So the micropay isn't the problem.

> you could provide a central service where people would buy credit to be used on many sites.

That central service is going to lock out many countries and regions as well as lots of people (minor, unbanked, poor, etc.) in non-locked out countries and regions. Payment is frigging hard especially on the international scale. This is every bit against freedom of information and strictly worse than Cloudflare.

> most people on HN only seem capable of attacking the few affordable solutions to that problem.

I doubt that many would attack those solutions if they actually worked well, but they don't. These "solutions" are a big part of the reason why the web gets smaller for me every day as more and more websites become unusable.

Cloudflare is like the TSA for the internet, I'm not convinced it needs to be as aggressive as it is. And yes I know websites have some control over how aggressive it will be but much like Reddit-moderation policy it's choosing the safety over everything approach, which hits enough false-positives on the edges to be a serious problem.

Cloudflare is very much anti-internet. And I'm a very security-obsessed person. Just like Reddit I believe we need to dial things a bit closer back towards chaos like a venn-diagram (safety)[x](chaos) there's a balance and I believe the internet is worse off when this balance is out of wack.

There might be some awful stuff on sites like 4chan but it also generated a ton of the memes that later filtered down into mainstream internet culture. Culture and innovation often happens in the chaos and fringes, which is an area I believe the world is becoming completely intolerant of in some attempt at idealism. But there are real sacrifices in between (ie, the mostly harmless stuff getting tagged as bad guys).

We need to be better at calming down and embracing the chaos, pushing back against FUD, and maintain a good balanced default. That chaos and flexibility is what originally made the internet great and endlessly promising.

Based on the various posts I've seen from Cloudflare founders on here I'm not convinced they are taking this problem as seriously as they need to be.

A comparison to TSA is flawed. Captcha is not a pass fail system, it is a score that is passed on to the web host and they decide what to do with it. Really any similar product to block malicious users would have the same problems, and the solution is to educate the website operators so they can avoid blocking legitimate users.

They do work well for the vast majority of people. Only on HN do I ever see people complaining about cloudflare/captchas/etc.

I don't dispute that they work well for the majority of people -- but the majority of people are not security-conscious.

However, I see people complaining about Cloudflare in lots of places other than here. The number of people adversely affected by Cloudflare is not small.

I see lots of complaints about Captchas in the “real” world, too. Not regarding the centralisation etc. aspect but more regarding how painful they are to complete correctly, but there are definitely complaints.

Regarding Cloudflare, a regular user will have no idea about what Cloudflare is and what they do. If something like the OP happens to them, they will just figure “the site is broken” and move on. So there could be a large hidden number of users who have suffered from overzealous Cloudflare blocking without being able to identify it as such.

> It's easy to shit on everything. Let's hear some real solutions.

My solution more and more is to just not bother with it. If a site is unreadable because I'm using uBlock and uMatrix, and I have to spend more than a minute or two tweaking things, then I just leave.

That said, I don't have any problem with Cloudflare. I'm much more annoyed by the overuse of *.googleapis.com. I'd love if somebody would setup a service that I could point my hosts file at so that googleapis.com silently went somewhere else.

I've been thinking about a local proxy that caches CDN assets. The first request to a cdn URL goes through, subsequent requests come from cache.

I think it would work fine with versioned libraries, fonts, etc. I'm thinking of setting up a container and squid config to achieve this.

Any obvious problems or alternative solutions?

Obviously enumerating the worlds CDN URLs would be a task. But I think even covering the most common CDNs would be a benefit.

I mean your browser basically already does this by utilizing cache-control and expires headers, which all CDN's are going to set

Something like https://decentraleyes.org/?

uMatrix is great for blocking 3rd party stuff globally in your browser. Outside of the browser, I rely on DNS blocking rather than modifying hosts files.

Wrote a little post about how I configured my blacklists and whitelists with AdGuard Pro for iOS.

https://www.calebyers.com/blog/dns-ad-blocking.html

The problem is that the narrative has been poisoned by Cloudflare and Google (for Recaptcha) - they both overstate the size of the problem, as well as the effectiveness of their solution.

In other words: when someone demands "real solutions", they're typically expecting a degree of solution that quite likely just does not exist at all, to solve a problem that isn't as severe as people believe, just because that's the bar that those companies have set in the public discourse.

This makes it impossible for well-intentioned people to 'compete' with these services, because whatever alternative is suggested (hidden form elements, a random VPS provider with DDoS mitigation, serving assets locally, etc.) is immediately dismissed as "that can't possibly be as effective / effective enough", even though it'd be perfectly adequate for the vast majority of cases.

The alternative and competitive solutions exist, and have existed for a long time. You don't need a 1:1 replacement for these services. People just often refuse to believe that the simple alternatives work, and won't even bother trying.

(For completeness, my background is that of having run several sites dealing with user-submitted content, including some very abuse-attracting ones.)

>because whatever alternative is suggested (hidden form elements, a random VPS provider with DDoS mitigation, serving assets locally, etc.) is immediately dismissed as "that can't possibly be as effective / effective enough", even though it'd be perfectly adequate for the vast majority of cases.

They are immediately dismissed because I don't want to pay a fulltime engineer to play cat and mouse with skiddies on the internet.

I think you're confusing what you wish was true with what is actually true. For instance, here was a post from a few weeks ago about how one annoyed user was able to take down a Mastodon instance until the admin gave up and put it behind CF: https://news.ycombinator.com/item?id=21719793. Bear in mind, if you're running a Mastodon instance, you're probably well-aware of the downsides of centralization and would only give in as a last resort.

CF has problems, but pretending it isn't solving a real issue that is nearly impossible to fix otherwise, especially for individual admins running a side project, doesn't help anybody.

> I think you're confusing what you wish was true with what is actually true.

And you are cherry-picking poorly sourced anecdotes to better suite your position.

A VPS with 100Mbps virtual adapter physically can't withstand DoS from single attacker with fiber connection (or equivalent of it). This does not have much to do with anatomy of DoS attacks, just simple math.

Cloudflare subsidizes their free users by giving a bit of bandwidth for free — the amount, that can be purchased from a decent hoster for several hundreds dollars. Of course, an attacker with several hundreds dollars can easily rent a botnet, that will demolish that "protection".

Huh?

"All Cloudflare plans offer unlimited and unmetered mitigation of distributed denial-of-service (DDoS) attacks, regardless of the size of the attack, at no extra cost."

https://www.cloudflare.com/ddos/

Do you know of an example of an attacker "easily demolishing" Cloudflare's free DDoS protection for a website with a few hundred dollars worth of botnet?

> Do you know of an example of an attacker "easily demolishing" Cloudflare's free DDoS protection

I can name dozens of websites, that folded under Cloudflare's supposedly flawless DDoS protection (at the time when they were still using it). Of course, the ones who fold are always websites themselves — Cloudflare itself is never affected, because when the DDoS gets particularly bad, they just detach websites from their CDN and expose it to attackers.

...so name them?

If I care deeply about my site staying up, a solution that’s perfectly adequate for the vast majority of cases isn’t sufficient. I don’t want to end up in the mirror image of the original author’s situation, where my site randomly falls down and I have no way to figure out what’s wrong or fix it.

Cloudflare is publicly traded under the symbol NET and has quite a number of institutional investors. A list can be found here: https://old.nasdaq.com/symbol/net/institutional-holdings

If all those companies are fronts for various parts of the US intelligence community then we're really screwed, I suppose.

That's... not how that would work.

For caching: Learn how to code. If your web page dies when there are only two visitors, then that's on you.

DDoS attack: If possible, the easiest solution is to just swallow the traffic. If that doesn't work you want to block all networks that allows IP spoofing. Then it's a wack-a-mole game. And if you have the resources, use any-cast and many co-locations. Or ask your ISP for help.

Hiding your server: Use onion address via TOR network.

SSL certificate: Use Letsencrypt

Edge SSL/DNS/CDN: Use a fast web server or proxy, like Nginx. With Cloudflare the connection to the Edge server might be faster, but time to first byte (on your site) often slower. So you get better bang for the buck by optimizing on your end.

Note that DNS by itself already have edge caching out of the box, for free! eg. if a user looks up your domain, it will be cached both at their ISP and LAN. So you don't need Cloudflare for DNS.

> Yeah, you can't really talk about downsides of Recaptcha/Cloudflare without also acknowledging the extreme amount of malicious actors and abuse on the internet.

What percentage of traffic on the long tail of 95% of smallest websites served by CF is malicious then? So that we talk in numbers.

I have run a number of small and medium websites (20 users per month up to 2 million). At least 50% of the traffic I see in my logs includes some sql injection or other mass script kiddie bs.

That's fairly black and white. Blocking an unusual user-agent because you "think" it might be malicious is another thing.

It might be a poor business decision, but probably not for the reason most people would think.

An unusual UA is unlikely to move the needle on top line metrics, but it is a distraction and a misuse of resources to play cat and mouse. (Unless your business would be materially harmed by someone scraping your data... in which case, you’re doomed anyway.)

I've looked at my logs, and obvious nonsense like POST or GETs with any search params on a website that only has static html pages wich should not generate these kinds of requests is about 1% of last 25000 requests.

Why care about such traffic? Blocking it seems like a pointless exercise.

I was responding to OPs question. Iirc, we discussed it and never implemented any blocking.

> Let's hear some real solutions

The old recaptcha which did not need js, did not serve you with unsolvable challenges, and did not refuse to serve you because you used tor/because you used the audio challenge too much.

Somewhat of a topic hijack and a naive question, but assuming Cloudflare is a government entity, wouldn't they still have to comply with whatever their terms of service / contracts with their users are? As they are a US company, barring illegality, theoretically they can't actually do shady shit without being in breach of contract right? They would also open themselves up to shareholder lawsuits.

If they were an actual part of government, sovereign immunity would be something that would have to be considered. In a nutshell, the government cannot be sued unless it decides to allow it.

The government has passed laws to allow itself to be sued under certain circumstances. The Federal Tort Claims Act (FTCA), for example, allows suits for a variety of torts.

I believe (but am not actually sure) that most normal business-type transactions with the government are covered under FTCA or other acts, so a breach of contract by Cloudflare-the-government-entity would probably be pretty much like a breach by any random non-government entity.

Still, if you were going to depend on that it would be a good idea to actually look into the details of the FTCA and other such acts and compare to the actual Cloudflare TOS.

I have no idea whatsoever how sovereign immunity works in the case of a corporation chartered under some state's corporate law (Delaware in the care of Cloudflare) that is owned (fully or in part) by the government. I'd guess that it could only possibly apply if the government owns enough of the company to have control.

Cloudflare is public, so we can probably not worry about that scenario. If the government actually controls them, it is doing it surreptitiously, and so even if sovereign immunity should be somehow applicable I'd expect that the government would not bring it up because doing so would necessarily bring to light their control.

What ever happened to proof of work protocols? I remember in the 00's they were being touted as The Solution™ to our spam/bot woes. Are botnets just so large that even PoW doesn't significantly affect them?

> Yeah, you can't really talk about downsides of Recaptcha/Cloudflare without also acknowledging the extreme amount of malicious actors and abuse on the internet.

But recaptcha bas been broken for years now by several different means. At this point, it is so broken it's almost a scam (and just another way for Google to get personal data from as many website as they can).

> How are you going to talk about the downsides of Recaptcha/Cloudflare without also acknowledging the extreme amount of malicious actors and abuse on the internet?

This is acknowledged by the original question

>> Also, how would your implementation differ to solve this issue?

To be clear, I agree with their comment and tried to double down on their point that it's a tough nut to crack. I improved my opening line.

> malicious actors and abuse

It's hard to consider simply viewing content to be malicious or abusive, no matter how automated.

I used to work for a data-scraping firm and very often we would accidentally knock many web sites offline when we pointed our crawlers at them.

I'd love to agree with you, but the crawler problem is 100x worse today than it was a decade ago

This would be much better solved with IP-based rate limits. And if IP-based doesn't work, then you're dealing with a DDOS, and it doesn't sound like this case was DDOS protection.

IP-based rate limiting is easily foiled via proxies, VPN services, tor, or botnets

And user agent string based protections are even more easily foiled, that's why I don't believe this can be plausibly counted under DDoS protection.

> Yeah, you can't really talk about downsides of Recaptcha/Cloudflare without also acknowledging the extreme amount of malicious actors and abuse on the internet.

Cloudflare have a long history of supporting those malicious actors, so it's not like the problem is unrelated to the purported solution.

Reply on news.ycombinator.com