Tell HN: Cloudflare Is Blocking Piped | Hacker News Reader

Tell HN: Cloudflare Is Blocking Piped

206 pointsassttoasstmgr120 comments

Lately I've been getting frequent "Error HTTP 451: Unavailable for Legal Reasons" thrown by Cloudflare whilst using Piped (a YouTube alternate front end used by Nitter).

However these errors are generated... in error. The page links to a DMCA complaint which lists about a half dozen unrelated YouTube (and Piped) links, none of which are being accessed when the error is generated. In fact, viewing the video on YouTube plays back fine. There appears to be a glitch in Cloudflare's URL filtering. It's been happening so frequently that Piped is often unusable.

View on HN

I also get CloudFlare now blocking my access to RSS feed MP3s for some podcasts. Once the almighty CloudFlare deems you a threat, your IP is burned. These days I can use less and less of the internet.

I really want to just see us get to the point where we don't have to rely on such services. I refuse to use them or any other for services I run, DDoS be damned.

I also refuse to use them for anything. A decade or so ago I spent 2 years barely able to access any site using cloudflare, won’t forget that

It goes to show the flaws of centralised services where you are not the customer. Not only is there no one to complain to, you can’t even take your money/traffic elsewhere as the competitors probably use cloudflare too

Cloudflare's default settings are very hostile to RSS feeds in general. They block these as part of bot blocking. Which of course is silly because these are intended to be accessed by bots. Even Cloudflare's blog RSS feed is affected by this.

I agree this is a problem, and we're actively working to fix it.

Specifically, there's a ticket in progress to improve how bot mitigation handles requests for certain types of static content (including RSS feeds).

Good to hear. It seems to me that more or less all static content should be exampt because it can be cached so serving it doesn't cost the origin anything.

Of course there are ways to bypass the cache with bogus URL parameters that make this difficult and some customers that are concerned with scraping even if that content is "static".

Cloudflare is making my live very difficult right now. Spammers are hosting websites using free domains, like .ml, .tk, so an unlimited supply of random domains, hosting them behind cloud flare which prevents us from easily getting the page content or blocking the IP for a period of time since the IP is shared.

Lots of spam hosted on cloudflare these days.

How do they make it hard to get the page content?

Is it easier/better to block by IP than to block unknown free domains?

No op, but

I believe that once you put your website behind cloudflare it's really hard, if not imposible to get content using requests. Don't know about scraping tho.

Also, I think it's better to block unknown free domains, because (public) IPs can have thousands of devices asociated with them. Once you block a domain, the "scammer" has to buy a new one.

We've seen similar things where spammers scrape our sites, put them up slightly modified and use cloudflare to block access to most of the web. They're obviously letting Googlebot through, but I've tried accessing it from dozens of countries and they're always straight up denied. I don't know what they're doing exactly, maybe it's an SEO attack, or they might be running ads and allowing that traffic to pass through.

If CF had a simple way to get (verified!) customer details, much of the crime using CF would go away while the pure DDOS-protection and CDN-usage wouldn't be impacted. Legitimate companies have their legal info on their websites anyhow, they don't care if you also can query CF about who they are.

You can get the originating IP via mod_remoteip, or its nginx brethren. You can block those IPs in your firewall, or via the Cloudflare firewall.

> or via the Cloudflare firewall.

Wouldn't that be a dream world for Cloudflare? "We protect spammers and if you wanna be as well protected against said spammers, sign up for our firewall"

Cloudflare might end a golden era of scraping, when it was trivial to scrape data from any site. Now Cloudflare helps site owners to make sure than only humans can read their contents manually. As more site owners switch to similar services, web will become less and less machine readable. No automated data processing, no archiving.

But wait, AI models will help bots looks like real humans accessing a site! They'll try hard to will fool the AI models that check if a site is browsed by a human.

Ha-ha, only serious.

No need for AI, browser can easily be automated and captcha can be solved using cheap services

> captcha can be solved using cheap services

call it what it is - you're using slave labor in a 3rd world country to solve rudimentary puzzles for you

It's probably not slave labor. It may be really poorly paid labor but if you had slave labor you'd probably use it for something profitable like construction like they do in the Persian gulf countries instead of solving captchas that people pay $3 per 1000 for.

this won't stop the overall trend, but it can help you get around cloudflare's effective scraping blocking (copying my comment from a previous thread):

If you're scraping with Python, try cloudscraper—among other things(!), it supports JS rendering (basically the bare-minimum check cloudflare does), without needing to run a full browser in the background. It's built on requests, so integration was pretty easy.

https://github.com/venomous/cloudscraper

JS rendering is not enough. Cloudflare monitors UI interactions and browser footprints to assess whether it’s a human or a bot.

trust me, I'm aware—cloudscraper can also solve cloudflare challenges, including turnstile

I mean it's Piped's decision to host their service on Cloudflare, no? No on forces them to use that service, so I don't see this as an issue with CF. They are not "the Internet", even though their marketing makes you believe that, thousands of large services run fine without routing their traffic through them.

I still dunno how people got conned that DoH aka. "tunnel your every DNS request to american entity that is required by law to spy on you on demand" to be the new "standard" for the browsers

in the US ISPs sell your DNS request data, compared to this Cloudflare seems an improvement

in other parts of the world ISPs give your DNS data to the not so secret police and compared to that Cloudflare is a huge improvement

in the parts where ISPs don't sell your DNS data you should switch to a different DoH provider

DoH is 100% a good thing. It makes surveillance of your Internet traffic harder, not easier. If you don't trust Cloudflare, then pick a different DoH provider that you do trust.

Nothing is 100% "a good thing", everything has tradeoffs.

In this case, you're moving the trust you put in your ISP or anyone who resolves your DNS queries to Cloudflare. Depending on where you are in the world, or how your threat profile looks, this might be good or bad, or degrees of good/bad.

That everyone is starting to tunnel more and more of their traffic to one single entity (Cloudflare or not) is overall not that good. But certainly not 100% bad.

> In this case, you're moving the trust you put in your ISP or anyone who resolves your DNS queries to Cloudflare.

Not necessarily:

> If you don't trust Cloudflare, then pick a different DoH provider that you do trust.

Have you submitted that to https://radar.cloudflare.com/domains/feedback ?

You can’t rely on cloudflare for infrastructure. They have proven too many times they will just drop stuff almost as much as Google will.

what's your alternative?

Honestly, if I wasn't a technical guy and I saw my channel and all my content on some piped.kavin.rocks or yewtu.be which aren't visually distinguishable from all the non-alternative-YouTube-frontend tube sites, I would assume someone's ripping all my content and impersonating me as well. I can totally see where the DMCA is coming from.

And even knowing the technical differences, one may want to dissociate with a stupid domain name like yewtu.be.

Edit: I showed https://yewtu.be/channel/<channel_id> to a content creator friend just now. Predictably, the reaction is "WTF, am I being impersonated? What should I do?"

Why is yewtu.be a stupid domainname? It's an "alternative" spelling of youtu.be that's easy to memorize and fast to type. I use a plugin to redirect to newpipe instances, but if I hadn't one I would probably use yewtu.be because it would suck to always type something like piped.kavin.rocks or invidious.pussthecat.org

I think that is a comment about alt right commentators online. YewTube sounds like an anti Semitic joke.

Jesus, people see nazis behind every blade of grass nowadays.

Pray tell, what is antisemitic about "YewTube?" Just that it's one letter away from "JewTube?" Sure, but if the creators intended to be antisemitic, why not just call it that then?

That seems like overactive pattern-matching to me. Yew is a plant.

Oh I even checked urbandict to be sure that it couldn't be this and all I found was the happy sound surfer make "Yew".

I think that's just a sign of the times that you see right-wing and left-wing comments everywhere even when it isn't there at all.

I thought about submitting this a week or so ago. Here's the link to the issue and discussion on it: https://github.com/TeamPiped/Piped/issues/1704

TL;DR: Apparently there's a Hong Kong dude living in Germany that didn't like his videos being on Youtube, so he sent DMCA takedown requests to Piped instead and Cloudflare did a takedown on the whole domain, which only appears if sent as a referral from outside piped.kavin.rocks (or using the redirect extension for firefox).

Close. Seems to me more like he didn’t realize that Piped is an alternative front end to YouTube, and assumed that someone had actually reuploaded his YouTube content elsewhere.

Yeah I realized it while sleeping and when I woke up it was too late to edit. I meant to say the takedown was likely sent to the abuse contact on the whois info for cloudflare's ip address. Could be his own content he's claiming or something he really didn't want public and just exhausted every potential avenue to send takedown requests to. Not going to assume either way, but that's likely how this started.

Is there a service like Cloudflare outisde DMCA vulnerable jurisdictions?

By now Cloudflare is more of an obstacle to the free web than it is helping. A centralized entity, whose scripts from randomly named subdomains you must allow to run on your machine, or be stuck at their obnoxious "checking your browser" page endlessly reloading, because some web dev decided to put their website behind Cloudflare. Cloudflare is one of the most prominent reasons for me to simply close the browser tab and leave the site.

I remember working on denial-of-service protection code for an embedded device.

One problem was that if the code was TOO aggressive in protecting from a denial of service attack, you could actually help an attack or be the culprit yourself by denying legitimate traffic.

I think this is what cloudflare is doing. They are imprecise and they are denying legitimate traffic.

I don't think that ever happens. If anything they are too lenient. Our own alarms kicks in way before Cloudflares DDOS protection is activated.

I don’t know… the so called free web is also a bot paradise, and like it or not cloudflare is actually helping mitigate it to some degree. It comes with a cost but maybe it’s worth it?

Cloudflare has played a major part in making VPNs suck, by providing a service that actively blacklists VPN IPs and selling companies on integrating the VPN blocker into their services.

It's probably true that some VPNs are used for nefarious stuff, but it's also lame that Cloudflare is such an anti-privacy warrior.

The web basically relies on bots to exist, search engines wouldn't work without them, Archive.org uses them to archive the web, etc.

It would be interesting to know what percentage of bots are actually nefarious.

I still remember when they posted some articles about those pages wasting time

https://blog.cloudflare.com/introducing-cryptographic-attest...

while they are main reason (in my browsing at least) the "verification pages" happen.

One thing I noticed was how cloudflare branding used to be pretty prominent on those pages, and now is pretty small.

I think they probably realized that maybe they don't want to be known as the reason these pages are showing up everywhere and inconveniencing legitimate traffic.

What scripts from random subdomains are you referring to? I know that from Cloudfront (Amazon's CDN), not Cloudflare. CF usually keeps everything on your domain.

The "checking your browser" isn't a default CF thing btw, that's up to the site owner and how paranoid they are (with or without reason). It's annoying me too, but we have sites on CF and practically nobody sees any checks when they access our sites.

Good to know that this page is due to the cloudflare customer! I am only seeing the results of that paranoia in my daily browsing and it sucks. I recently had to ban Gitlab into its own browser profile, because with my previous main profile settings, it simply wouldn't let me log in. I am treating it from now on as contagious, because of that "checking your browser" bs.

(I did write a support request message to Gitlab, but their support clearly sucks. What do I know what kind of subscription my employer has? I don't care! They are paying for me, so Gitlab should offer a modicum of support, if I cannot even log in on their shitty site any longer, because of their changes. But they stonewalled with something like: "We need to know your subscription level blablabla before we can continue the process." kinda automated e-mail. Well, duh! Check your friggin database for my subscription level. Oh but then you would actually have to work. Ah that's a problem of course. Better stonewall a paying (paid for) customer.)

On a theoretical level, a service like Cloudflare is the most terrifying entity on the Internet I'm aware of. They've accumulated an insane degree of insight into the traffic flow of the web (since their entire service is essentially acting as a HTTPS middle man), and their business is offering protection against bot spam that could ruin most websites. Even if they aren't operating the bots themselves, they're essentially displacing the bot problem to the unprotected websites. Like the overall shape of this operation is something the cosa nostra could have cooked up in the 1970s.

However, being on both sides of this, both operating a bot for my search engine, and operating a web service that is aggressively targeted by bots. They're not actually bad to deal with.

The big unanswered question is how they'll manage to stay good given the obvious incentive of abusing this setup. Maybe this CEO has a moral backbone, but will the next, and when they're acquired by the Meta-Amazon-Alphabet group in 15 years, will they still stick to these principles?

He has shown time and again that his backbone's strength depends on how loud the public noise is. Kiwifarms most recently. You can dislike them(kiwifarms etc) and there is a case for them to be taken offline imo, but it is the governments job.

Exactly what you do _not_ want protecting the neutral internet. They've done better being neutral than some might have, but that's in reality more insidious because clearly there are points they will bend on and those points will change over time and almost certainly continue to erode.

ALL big tech companies have the same setup. There is nothing unique with Cloudflare. People are just talking about Cloudflare cause it is accessible for free and they sell it as a service.

Internet security has, in my experience, always been about "being just hard enough a target the bad actors decide to go torment somebody else."

It was true twenty years ago too, the only difference I can see between then and now is that you can outsource that task for a (relatively) small amount of money if you want to.

Then again, the last time I dealt with a site under DDoS, something in their stack was leaking the underlying IP (never did figure out what) but it turned out that "finding a provider who'd sell them a decent sized server and charge them for the bandwidth" was perfectly economical for their use case because their haters' firepower was insufficient compared to their revenue.

(I'd love to be less vague here but I'm sure readers can see the obvious professional ethics issues with doing so)

I'm surprised you're handing incoming requests from everybody. We only process the CloudFlare ones and drop the rest.

You can fill the pipes to the server(s) you're targeting, it doesn't have to be application layer.

These days, Cloudflare lets you serve your origin via a tunnel from a host that doesn't even have a public IP.

And if you run that in a cloud, the NAT isn't your problem -> your attacker will have to DoS that cloud as a whole.

That's an extremely smart approach that I sincerely doubt the site operators would have been capable of dealing with.

Part of the art of consultancy is, sometimes to my great annoyance, optimising for "within the customer's budget" and "within the customer's capacity to maintain it after I'm no longer involved" over "best possible solution."

Plus in this particular case I was working pro bono because (a) I quite liked the site in question continuing to exist (b) a Shadowcat alumnus asked me nicely (c) I take great pleasure in ruining a griefer's entire week. So lightest possible touch was strongly indicated.

The end result was not remotely clever, but it's been in production for a while now and has not to my knowledge caused financial or uptime issues, so I'm going to call it a win even if the inelegance of -how- I won continues to irritate me ;)

Right, "finding a provider whose reaction to an aggravating quantity of incoming packets was charging money rather than throttling the connection" was basically the load bearing part of the solution here.

Fortunately, while said quantity was indeed aggravating, it was low enough that the cost was financially and logistically less than trying to do something more elegant.

Sometimes brute force and ignorance is, in fact, the right answer, and I don't have to -like- that being true for it to be true.

>The big unanswered question is how they'll manage to stay good given the obvious incentive of abusing this setup.

Why do you think they're still "good"? CloudFlare has chosen to abandon sites that held free speech (abhorrent speech, but still free speech) while still protecting forums upon which credit cards and methamphetamine were listed for sale on the front page.

To me, that's not a sign of a "good" actor.

Free speech doesn't exist within the context of a privately held website.

Free speech is an ideal, not just an amendment.

but a private entity (person or corp) does not have any obligation to protect ideas they find abhorrent to be considered on the side of "good".

That's the point. They find free speech abhorrent but consider selling dangerous drugs to be acceptable.

So many people find them abhorrent because they represent different values.

Legally no one has any obligations here.

Agreed, but it's a strange value system that says, "Dealing meth and stolen credit cards is okay, but having a web forum that makes fun of people is not."

I always figured that the main thing Cloudflare protected against was DDoS attacks, not bots (DDoS may be caused by bots, but with significantly different outcomes -- a single bot in and of itself won't take down a website)

RE bots: TikTok has incredible bot protection that comes from engineering (webmssdk) instead of network-based filtering. I'm not even sure if they use Cloudflare.

Cloudflare doesn't even really protect against DDOS. Sometimes taking your website off Cloudflare is the only way to stop a DDOS attack. That's because you can't stop something like a level 4 ddos attack by blocking the IPs in raw prerouting iptables, because if you did that then you'd be blocking Cloudflare's IPs. The only option Cloudflare really provides you is pressing a panic button that forces everyone who visits your site to view a captcha, when it's really so trivial to just run the iptables commands using a token bucket algorithm. I know because I run a website on a 2 vCPU VM that gets DDOS'd all the time. I've had to block over nine thousand malicious malicious IPs so far. I tried using Cloudflare in the past for their protection services, but it made me (1) defenseless against bad visitors and (2) made good visitors angry at me for the captchas.

How did the attackers get your origin ip to begin with? I thought cloudflare was supposed to shield it at the DNS level, and in theory your origin should be dropping all connections not coming from an authenticated Cloudflare proxy?

They weren't able to talk to my origin IP, because when I was using Cloudflare, I blocked at the firewall all IPs that weren't Cloudflare. The problem is that they would DDOS my server through Cloudflare. And because the traffic was being proxied, I couldn't block the attackers without blocking Cloudflare. Unless of course I wanted to fill out a form on their website 9,000 times. It's an awesome website by the way. I love their workers and r2 products. But Cloudflare honestly isn't that good at DDOS protection. These attacks were so bad that Cloudflare would start showing NGINX error pages before my web app even went down. Cloudflare should be paying me to protect them, rather than the other way around.

Do you have a support ticket # you can email me w/details (pat at cloudflare)?

We take every reported false negative as an opportunity to improve our DDoS mitigations, and these reports are very helpful.

As of a few weeks ago, you can now report FNs/FPs for Bot Mitigation directly in the dashboard, and we'll be expanding this pattern for use with DDoS Mitigation as well.

They do both. Ddos mitigation happens at the network level, while bot protection uses a combination of whitelists, blacklists, behavioral heuristics like mouse movements, login state, and captchas.

Crawling websites behind Cloudflare can also be problematic if they (CF) decide that your bot doesn't fit their definition of OK. This is problematic for new search engine entrants and a multitude of other services, particularly given how many sites now live behind CF.

Years back their DNS service also stopped honouring ns_t_any requests (for reasons of DDOS amplification apparently).

I do tend to agree with you about centralisation, gatekeepers particularly.

You can't run around and crawl other peoples sites. That time is long gone.

Not sure if you're being sarcastic!

In the end for any scraping they're just raising the barrier of entry. Automated browsers, residential proxies, captcha services just make it more involved for those determined to hit a URL successfully.

Not necessarily a bad thing, but the line and grey area and the definition of a 'legitimate' request varies, and one entity as a middle-man deciding that is less than ideal.

Not sure where you coming from but let us go back 10-15 years when there was an open market for commercial crawlers and IP ranges to be used for it. You sold shoes and scraped all other competitors for instance. That era is over.

For legitimate interests today including search engines and services for price comparison that data is often provided for free.

There are design patterns used today that does among other things provide incorrect prices to scrapers.

Scraping is illegal in most western countries btw.

>For legitimate interests today including search engines and services for price comparison that data is often provided for free.

Can you explain that further in the context of search engines, new or existing, need to crawl websites and Cloudflare are a barrier to entry? You seem to contradict yourself.

>There are design patterns used today that does among other things provide incorrect prices to scrapers.

If you say so, and hopefully they do it 100% correctly.

I never really understood Cloudflare's intent, because from the marketing material it seems that you get DDOS "protection", free TLS certs, everything in a monthly package, affordable, bla bla bla.

But from some basic calculations I get that R2, Workers and egress bandwidth beyond a few terabytes costs just as much as Oracle cloud / Alibaba.

But what I dislike the most is how little control you have over what's going on there. Like: If you haven't setup TLS on your webserver, why do they allow unencrypted traffic to flow between the server <-> Cloudflare and encrypt it to the end users and pretend that is secure?

Why can't they forward all my server's headers? Why <XYZ> ?????????

Read some horror stories on Hackernews and you'll quickly find out what their "unmetered bandwidth" really means. You get very little if any transparency about the pricing, which I would except from tiny cloud companies, but this is supposed to be a major one!

The two actual whys you have posted are settings you can change in the cloudflare config.

> If you haven't setup TLS on your webserver, why do they allow unencrypted traffic to flow between the server <-> Cloudflare and encrypt it to the end users and pretend that is secure?

I Really Can't Think of Any Reason

A few TB/mo is quite enough for a lot of smaller companies, and DDoS protection is something that a smaller company can see as a pretty valuable thing. A CDN with thick worldwide presence does not hurt either. So using Cloudflare is a no-brainer for a smaller business, especially with the prices they offer. Not using Cloudflare means either buying separate DDoS protection (likely offered by your cloud provider), or risking an extortion attack.

Some competition exists, but it's both more expensive and less reliable and convenient.

I agree that it is difficult to know exactly what you are paying for but they are very affordable.

I think the ability to put TLS in front of a non-TLS'd website comes of a few properties:

1. It's probably better than nothing. 2. It's a legacy thing.

A company like Cloudflare has to make a choice - how frequently do we break users who've set up their site in a way that is no longer in line with security best practices? It looks like the decision they've made is to break infrequently. Certainly the site I set up in 2014 when their free TLS was new still runs, and I haven't made changes.

I believe that you can set up strict TLS between Cloudflare and the end host if you choose, but it's up to you. I think in that instance, your 'little control you get' is actually more control, no?

And, if you look back even a few years, TLS was both uncommon and expensive. Cloudflare was a pioneer by offering free TLS certificates in I think 2014 (only 8 years ago!). LetsEncrypt started in 2015 and was niche for quite some time. I think even now you can find Linux distros preferring to ship their data over HTTP with GPG-keys recommended for the security. Of course in 2022 even simple sites should be TLS'd, but Cloudflare's existed for a while.

And, TLS to the client but plaintext from CDN to site is still better than cleartext the whole way, because it (generally) stops the ISP from snooping on its customers.

    I think even now you can find Linux distros preferring to ship their data over HTTP with GPG-keys recommended for the security.

This isn't really to solve the same problem though. The GPG key thing is so you can use mirrors for hosting that are distributed but still trust the package came from the real source. TLS termination of where the packages are retrieved is separate.

Yes, the gpg piece provides that functionality nicely. However, it’s exceedingly common for the mirrors to not be provided over TLS for cost reasons. Netflix switched to serving video over TLS for no other reason than to promote the usage of TLS (after a lot of custom engineering (pki on cpu, crypto on nic iirc?) to reduce the overheads of doing this.

Can't you use "Strict Origin" cert on Cloudflare? Here is a pic of my settings: https://i.imgur.com/aHQ1U1L.png

Sorry if I am missing something here. Cloudflare gives flexibility to their customers. That seems right.

Cloudflare enterprise is pretty transparent if you've gone through the sales process. They tell you exactly what the limits are. For average person, on free plan, they are not obligated to provide details of where the limits are. That's no different than BackBlaze unlimited storage plan.

> But what I dislike the most is how little control you have over what's going on there. Like: If you haven't setup TLS on your webserver, why do they allow unencrypted traffic to flow between the server <-> Cloudflare and encrypt it to the end users and pretend that is secure?

I don’t get the issue here. The traffic between client and Cloudflare is secure. SSL is terminated at Cloudflare. You can choose to have end to end security if you want.

If you set up your own frontend that terminates SSL, but choose not to secure the traffic to your backend, the end client will still see the connection as secure.

One of the reasons why it became like this is because there is no protocol that would allow a host to request blocking traffic from other host on upstream provider (so that malicious traffic is blocked close to originating network). If there was such protocol, site owners could protect from attacks themselves, but without it you have to use Cloudflare unless you are Google scale with channels wider than attacker's.

See if changing the URL from piped.kavin.rocks/watch?v= to piped.video/watch?v= will work. I've never had the "Unavailable for Legal Reasons" error when using the latter domain name.

Additionally you can set up a permanent redirect with a browser addon like the Redirector to always be sent from piped.kavin.rocks to piped.video.

CloudFlare again.. Offering their service to crime forums, credit card fraud shops and phishing websites, while making usage of Tor and VPNs nearly impossible or atleast a pain.

Coupled with the hypocrisy of an open web and freedom of speech, it makes CloudFlare arguably one of the worst threats to the web as we know it.

Whereas the freedom of speech ala Cloudflare stops as soon as it can generate cheap PR, because then a website is quickly blocked after a few media reports.. or in case of Piped as soon as the content mafia is complaining.

There is nothing in Cloudflare that blocks anything like that by default. Site owners decides what to block. The problem with VPNs and TOR is that there is a lot of rouge traffic from these services. Also, there is no feature that blocks VPNs in CF. Some get blocked for not coming from consuming ISPs but more commonly whole ASNs are blocked if the majority of the traffic is bad.

rogue traffic, unless you mean pink powder

Why would anybody in the right mind centralize his/her infrastructure?

I doubt that people actually need something like Cloudflare.

You can use shit slow language with fat framework and just put it behind CF and run half decent, that's why people use it

But you can also use a fast language with no framework, but host it on a 5€/mo VM and put it behind CF and it will run half decent.

Because they save us ~20 thousand a month in bandwidth cost.

We got 10k visits in a single day. Cost of data transfer: zero

Really zero, like non of the visitors was hitting the original servers? That would be impressive then. And you should consider to make money with delegating the traffic, not give away the traffic for free.

I mean, I did not went into the rabbit hole of checking thoroughly, but in cloudflare it says we served 8gb and aws says we served just a few megabytes.

You configure to ignore everything, even the url querystring, and worst case scenario, they serve your site from an internet archive snapshot. You can literally power off your server and the page stays online

im non binary

I couldn't run search.marginalia.nu without it. I've seen up to 50,000 bot queries per hour (and peak out at about 500 human queries per hour). I don't have the hardware to cater to the bots. I also don't have the money to buy the hardware to eat the cost. The options are hide behind cloudflare or shut down the service.

It's not about traffic costs, but processing power.

Can you please explain what exactly bot were doing? What was their goal? Yes, I've seen bot scraping sites, which is expected. But what queries bots were doing towards niche search engine?

Search queries look like spam, like the sort of spam keywords you will find in comment spam. "Free cialis 50mg online pharmacy near me"-type stuff

Best guess is they're gambling I'm backed by Google's API and trying to poison their suggestion data.

Sorry I don't follow. Could please elaborate. You mean bots do query 'cialis' to get an ad-sense ad, while they are the same guys benefiting from ads shown? Or what? I genuinely want to understand the problem and most importantly the motivation.

I don't understand the motivation either, but I think what they are attempting is to make e.g. typing cialis into Google suggest specific queries like the one i showed, which may be so overspecified they provide the spammers' links.

That's my theory anyway.

Made an account to say “their”. We are talking about cloudflare and cloudflare users. Their gender is not relevant in this conversation. At the end of the day we are people.

Companies centralize their infra - be it on AWS or some VPS provider.

Was at a startup that paid 5k/mo for Cloudfront and moved to Cloudflare and paid just 200/mo. DNS performance improved as we switched over to Cloudflare as well. Saw a decrease in bot traffic. No complaints about usability or being blocked.

So yes, Cloudflare was useful and helped saved $ for us

LibRedirect is working fine with Piped for me. I think I hit some blocked URL but it's trivial to remove it from the list, most mirrors are definitely working.

I used to love Cloudflare but their argument for free speech absolutism went out the window when they started making judgement calls about which sites to block and which to keep. Now I'm just disappointed but not surprised. Will probably move off entirely once Tailscale funnels allow for custom termination CNAMEs.

If this particular instance is them getting DMCA'd then it's not really their fault, but I'm confirmation biasing it with a pattern I see of them making more and more judgement calls about what to host and becoming more like a standard 100% profit-driven megacorp hosting provider.

Not to mention that their priorities when it comes to blocking decisions seems odd. DDoS-for-hire (stressers), piracy, ISIS support-forums, revenge-porn etc. are all fine because free speech. But a forum supporting nazis, an imagebord with lax moderators, and a forum archiving illegal/insane activity that people post online are all nuked because.. ... reasons..

"I know that Cloudflare is legally required by the US government to abide by the DMCA, but this supports my theory they are censoring the web more and more on purpose!"

?????

No I'm saying this case explicitly does not support my argument, however there are enough cases of them making judgement calls that this independent pattern has formed and it's easy for me to fall into the trap of confirmation bias. My initial comment wasn't clear but it doesn't let me edit now.

And cloudflare again!

Those guys... not to mention their pesky "browser verification" which is does not work with noscript/basic (x)html browsers.

It's the site owners who enable this; they are just not interested in users who run noscript or any other non-standard setup.

I’ve been running into this exact problem in recent weeks. Switching to piped.video is the workaround I’m using atm.

This is the same company that has repeatedly gone to the mat to ensure Nazi’s and targeted hate campaigns remain active online. But this is where they draw the line?

They have on multiple occasions had long and public campaigns talking about how important it is to fight censorship in all its forms except a random DMCA troll in Hong Kong?

I don’t think Cloudflare really love “free speech” as much as they pretend in their public messaging.

They weren't hosting anything..

You know exactly what I mean here. I’m going to update the post though.

Listen, the stuff is on their hard drives, being served by their servers through their public IP addresses. I don't care whatever backend method they use to update their cache from some other origin, by all accounts they are hosting and serving it.

It's not on their hard drives.

Why don't you go and complain to Telco providers, and undersea cable infra for forwarding pro-nazi bits.

They’ve never gone to the mat to defend free speech. They make a public statement indicating the discomfort they feel blocking content and then they censor it a few days later.

They have in this case https://twitter.com/stealthygeek/status/1485731108822077443 I don't have the case docket link easily available, but it was referenced somewhere around that thread.

They draw the line on a legal request. You don't want them to break the law, do you? There's a difference between pulling content off due to your disfavor of the content itself and legal requests to take it off.

Reply on news.ycombinator.com