Do not use cloudflare if you value a free and decentralised internet.
I can't afford to serve the 98% that is bots. That's about 10 search requests per second, most of them are search queries aimed at poisoning the query log of my search engine, which doesn't exist. But I think they're just scraping for opensearchdescription definitions and spamming every endpoint they can find in the hopes it's wired up to Google or Bing.
Cloudflare are the only ones that seem to offer some sort of (affordable) mitigation against this. They're hiding behind a botnet so rate limiting and ASN/IP blocking does all of bupkis.
The other option is to shut down my website. I believe that world is a worse world than one where stuff gets routed through Cloudflare, although I'm fully aware that this is far from ideal.
What I do to help those who want to avoid cloudflare is offer API access, which means I give you a token so I can rate limit you. Also means less anonymous of course, but it's at least free from men in the middle.
What benefit do those bots get from "poisoning the query log"? And what benefit do they get from "spamming every endpoint ... wired up to Google or Bing"?
So what's happening here could be one of two things:
Link injection, where by spamming an endpoint with a URL they hope the URL will end up indexed and displayed somewhere on the site, for a search engine to pick up later. Often with some keywords in the link text, etc. The idea is to boost pagerank.
The other is even sillier: they try search for their own sites a tonne on third party search things that probably query a "real" search engine, to try artificially boost the supposed popularity/keyword associations of their sites, boosting pagerank.
A lot of blackhat SEO stuff us bunkum and snake oil.
Every so often they figure out a working method to exploit search engines to boost pagerank, and then it gets fixed. But they never stop trying even the most obsolete methods as the automated scripts are what they have available.
As described on the Security Now podcast, this is now part of "internet background radiation"
The only place that filter exists is where people actually communicate openly about their strategies.
The fact of the matter is they're basically spamming very specific search queries and not really looking at the results.
But most larger search engines store and analyze search queries to for example produce typeahead suggestions or error corrections. You can conceivably poison this data to manipulate how a search engine operates, perhaps directing users to your website or affecting ad auctions.
I think the benefit of not directly spamming Google or Bing is that they're harder to identify.
But this is a publicly available Internet search engine we're talking about. There's slightly more processing involved in processing a search query than loading a static file off a filesystem.
With TLS it’s arbitrarily easy to overwhelm a target site that’s hosted on the smallest instance types at $provider due to the compute requirements of terminating TLS. Thankfully Cloudflare + LetsEncrypt makes it economical to host a personal site with good security without arbitrarily and suddenly high bills, constantly high bills, or my site arbitrarily disappearing off the web.
Bad actors poisoned the well (thanks China), and now here we are. I, for one, appreciate greatly what Cloudflare has done for the indieweb.
https://www.bufferbloat.net/projects/codel/wiki/CakeRecipes/
Yes. I could.
This is one of several possible solutions available to me as an alternative. All alternative solutions to this problem either:
1. Are more expensive to implement, for various values of "more expensive".
2. Are more complicated to implement, for various values of "more complicated".
3. Are less resilient to the problem space, for various values of "less resilient".
In some cases, they are all three. The proposal you made happens to be all three.
If I were to do things all over again from scratch /today/, I might do things differently, simply because I'm more well off and have a more stable physical location than when I set up the last iteration of my personal site (which is still running), but none of the possibilities I can think of are strictly "better" than what I have already done, leveraging Cloudflare.
I can't host a website for my home internet without the risk of getting black holed.
Can't ask the website for my home internet because Comcast won't let me.
Providers will charge you out the ass for bandwidth if someone ddoses you.
The internet just isn't a sustainable place for self run websites now,
I wish residential providers would support a second level of quota management. Assign 90% of my bandwidth to this website, then take it down for the month without completely killing my internet.
So this isn't insurmountable. You don't need a centralized provider for protection from decentralized abuse.
Having my website instant-killable "for fun" by any bored script kiddy living in a country thousands of km away from me is not a particularly interesting proposition.
There's also tons of sites that get the "HN hug of death" when they're posted here (or Reddit, or wherever else that generates a surge in traffic). Tragic to miss out on all those potential hits and easy to prevent with Cloudflare free.
You can dislike Cloudflare policies, but they do provide a valuable service and often completely free of charge. That's exactly why they have such a large slice of the internet under their domain. People don't just opt in for no reason.
Sure. But I was originally responding to a comment that claimed the key reason to use Cloudflare was DDoS protection. If the traffic is real, it's not too unreasonable to pay for it.
Using any service comes with costs and benefits. Any reason you have to use a service irrespective of their behavior is just a blind spot.
Why would you expect customers to migrate someplace else at all if there's no alternative?
Of course, having DDoS protection on the machine being flooded is a non-starter.
DNS would have to be highly available as well.
And it all needs to be maintainable by someone who's main job is not maintaining infrastructure, and doesn't have the expertise. Assume hiring someone to do so is not in the budget.
DDoS management is harder. You just kind of need to assess your risk and take appropriate steps given your risk. If you're likely to attract real, determined, attacks, you need a good solution.
If you're going to just get bored idiots that control a lot of bandwidth, but don't have a real beef, accepting that you'll get null routed for some time when you get attacked is really the least hassle option. Get as big of an incomming connection as you can justify, make sure you can discard bogus packets at line speed, take steps so that you don't amplify repsonses, and cross your fingers.
Be aware that moving to new hosting while under attack can be difficult (most hosts do not like customers that attract DDoS, especially new customers), so if you do move, communicate the current situation to the new host beforehand.
If you have important services (APIs or what not), offer them on hostnames other than www or your apex domains. Idiots flooding random people for lols really like to hit www, and ignore your other hostnames.
Free basic services arn't a loss leader, they're paid for by the enterprise contracts.
The loss leaders are always paid for elsewhere, or at least that's the plan.
Not saying you don't. Saying your experience is not everyone's experience here.
What would need an experiment to prove that? You would just have the proverbial receipts
Or do you have proof that they’re actually ddos in people?
Step 2: Enable cloudflare, paid tier, hang around with it turned on for a month or so
Step 3: Simulate some usage traffic
Step 4: Cancel your cloudflare contract
Step 5: Watch your unknown site get ddosed
Coincidence? Suuuuuure
Cloudflare itself doesnt need to ddos you, plenty of ddos for hire outfits out there that will happily do it.
As to who hires them, well that's between them and their customers. But there are some things that are not coincidences.