Turns out half the internet has a single-point-of-failure called “Cloudflare” | Hacker News Reader

Turns out half the internet has a single-point-of-failure called “Cloudflare”

easydns.com

972 pointsStuntPope385 comments

Cloudflare is horrible for blind people.

Screen readers, the programs that use synthesized speech to tell us what's on the screen, cannot read images. Good captchas usually have audio equivalents (which come with their own set of problems), but this one doesn't. If you're blind and flagged by Cloudflare for some reason, you're cut off from accessing half the internet, potentially critical banking/governmental/medical/communications/educational services. We rely on the internet way more than our sighted peers, so this is very important. This has recently happened to me on a few sites, fortunately not critical ones, but it was not a pleasant experience nonetheless. CF engineers, please fix this ASAP. I'm surprised there still isn't a huge lawsuit over this, as this is clearly violating all sorts of laws.

I use audio captchas. Google will usually only let you do 2 or 3 before banning you and making you do image-based ones. I'm pretty sure the button is just there to make it seem accessible.

I had exactly the same issue. I am surprised how neither cloudflare nor google have been sued for making most of the web inaccessible to people with disabilities.

Shouldn't that liability be with the operators of websites that use those captchas if they're required to be blind-accessible? If they get sued, it'll apply pressure up the supply chain.

I agree. You don't sue the contractor because they built to the specification you provided.

At the same time, if Google is advertising Recaptcha as accessible and it's not really, then they need to be held accountable for that, because that has a real impact on huge swaths of the internet, and especially for sites that try to do the right thing and find out Google has screwed them.

> You don't sue the contractor because they built to the specification you provided.

In the world of licensed contracting you absolutely do. If you provide a contractor specifications that result in a code violation, they have to tell you no. If they don't tell you no then you, the non expert, are reasonable to assume what you asked for is acceptable. If they actually build the thing, they are liable for the violation of code.

The last thing we need is to hand out more excuses to evade accountability and responsibility for making the world worse.

Yeah, contractor is a bad metaphor. I started with Home Depot, but that didn't really fit either. It's more that you bought a ramp that's designed to fit over a two step rise. Was it advertised as accessible in your state/county (which will likely have specific gradient and handrail rules? If not, you should have looked. If the salesperson assured you it was, well that sucks, but I'm not sure you'll get more than your money back.

> The last thing we need is to hand out more excuses to evade accountability and responsibility for making the world worse.

I'm definitely not advocating that. I'm more advocating that you can't leave all this to someone else entirely. Even if you pay someone for a turn-key custom website and make sure it's accessible, there's no guarantee it still meets standards a year from now, just as if you pay a contractor to build a ramp for you that perfectly meets standards. Things change, and people need to pay attention to the things that matter to them to make sure they follow the law. Whether that's occasionally checking it yourself or paying someone else to do so (and thus ensuring it happens), it still needs to happen.

I used to feel and think this way. But recent events, and the trajectory of the internet over the last decade has changed my mind completely.

We must stop giving excuses to the massive centralization, to the enormous companies that step in and rent-seek what is supposed to be a distributed system.

The most powerful tool we could have to get a proper internet back is the simple concept of accountability. If I was a bakery and I made a cake with poison because I was specifically asked to do so by a customer, I would still be accountable for the dangerous thing I made.

Stop making excuses for these companies with basically infinite money. They are the last ones who need it. What we all need from them is accountability.

This also goes for section 230 protections. We do not need Twitters and Facebooks and Hacker Newses and other such companies that would supposedly not exist without section 230 protection. It's clear now, in retrospect, that accountability is far more important than the license to grow enormous without any responsibility for the toxicity your giant bloated corpse of a business unleashes onto the world.

Anyone making a captcha product can and should be expected to make it work for anyone. And if there is basis for lawsuit, it should be the maker of the broken thing that faces it.

Hell, sue them both. Neither entity deserves a free pass. They both had a part to play in the exclusion.

If we want to regulate the captcha industry there should be a law passed specifically regulating the industry. Blindsiding the industry with judgments against them when there are no laws about it will decrease confidence in the rule of law, which will hurt the economy as businesses lose confidence in their ability to predict the future.

Rules and laws are good tools and, I understand how law works in the most superficial way but, I fail to understand normalization of ignorance of some required features with the "nobody forced us to do this, so we just didn't care".

As human beings first and programmers later, we should understand that people with disabilities exist and assistive technologies need attention.

Why, as decent human beings, don't we do something nice without the push of laws, for once?

I agree we should do nice things without the push of law. But ztjio was advocating for the push of law without following the process of creating a law, which I disagree with.

There are already laws regarding accessibility.

Are Captcha companies violating those laws? If so, then yes, enforce them. If not, then either change the laws or leave the Captcha companies be.

> Are Captcha companies violating those laws?

I would argue that this is the case, yes.

> If so, then yes, enforce them

If only I could.

This does not really make sense, it would be perfectly fine if two products existed, a cheap inaccessible captcha and a more expensive perfectly accessible one with companies offering both of them to users.

Fine in what sense? In my opinion, it is not fine to block disabled users out of webpages just because it costs a bit more to let them in.

That is not happening in what I described. A webpage could decide to offer the cheapest working captcha as a default with a "Are you using assistive technologies?" / "Do you need help?" link to a more accessible service.

I don't think that is will be common in the wild, but it is an argument against regulating captcha providers rather than websites.

Oh, I see what you mean. There may be a market for "accessible CAPTCHA", that's audio-only.

Have you tried our accessibility option? It's perfectly accessible https://www.hcaptcha.com/accessibility

As sad as it is, there seems to be a CAP situation when it comes to captchas. It's accessibility, security or privacy. Choose two.

You can go the Google route and choose accessibility and security, do massive user tracking, and don't even show CAPTCHAs at all for normal users. HN users probably see a lot, because they use some anti-js or anti-tracking stuff, but normal users don't.

You can go the Cloudflare route, requiring users to solve visual challenges, sacrificing accessibility, but keeping security and privacy.

You can also implement audio CAPTCHAS, which are easy to solve for robots, get accessibility and privacy, but less security.

> You can go the Google route and choose accessibility and security

This is not the case. After just a few captchas the audio challenge will be locked off no matter what browser you use.

> and don't even show CAPTCHAs at all for normal users

You see them in firefox from the start as well as on chrome after around the 2nd or 3rd captcha, this is with the default settings + ublock origin on both browsers.

CloudFlare doesn't use Google ReCaptcha anymore. They use some Blockchain Captcha thing now.

Thank you for sharing your experience.

Cloudflare switched to using hCaptcha a couple months ago, I think and I only just noticed that they do not offer an audio-based captcha.

However, hCaptcha integrates with Privacy Pass[0], and you can top up your tokens by solving a captcha at [1] and [2]. What you could do (and I realize this is far from ideal), is getting a sighted friend to solve a couple of captchas so you have enough tokens to last you a couple months.

[0]: https://privacypass.github.io/

[1]: https://captcha.website/

[2]: https://www.hcaptcha.com/privacy-pass

This is such an out of touch dev response.

The problem is hcaptcha, the solution is stop using hcaptcha. Not placate and evade with work arounds you wouldn't even suggest to non blind people.

Also this ask a friend solution is like telling someone in a wheelchair to ask a friend to help them up those steps instead of just installing a ramp.

You don't get to take over half the internet and just ignore social responsibility.

Solutions can’t be implemented instantly; a work around is needed until that ramp is installed. As much as the lack of audio solution sucks, you can’t expect half the internet to just give up on a piece of technology overnight

Why not?

When you move fast you tend to break things. When half the internet relies on you you want to avoid outages.

> you can’t expect half the internet to just give up on a piece of technology overnight

People call for this all the time when a major security vulnerability is discovered. The difference is in how the community views it.

Cloudflare should weigh harm of blocking access to the Deaf community warrant the harm of removing a captcha that doesn't support them. They should have done it when they chose a captcha that doesn't support the Deaf community.

As technology comes to mediate every avenue of life, we need to recognize that technology only has value in its positive effects on people's lives. A security vulnerability is bad because owners may lose control, property may be lost, crimes may be committed. Usability vulnerabilities can deny services essential to their users. You're totally correct that there is no magic solution, but to say that we know which imperfect solution is preferable is incorrect.

No. The ramp needs to be built when the thing is built. Accessibility is not an afterthought.

I mean, yes, in an ideal world that's true. Yet, here we are.

It's not possible for a dev to go back in time to before hcaptcha was created or when CF decided to switch to them to create said ramp, so until it's built, workarounds are the only real thing a community member can offer someone in the short term.

It's not like trying to find the Fountain of Youth or something. It's a company with billions of dollars making it impossible for certain people with disabilities to access the Internet. Let's not pretend that it just has to be this way. A world where half the Internet can't be broken by one company should be the baseline.

Nobody is saying it has to be this way. As I said, in an ideal world, accessibility happens when development happens and is not an afterthought. But we don't live in an ideal world, and it's clear here that it wasn't thought out when they did the switch, or other forces caused the switch to happen without this piece in place.

So, with that in mind, knowing where we currently are, not where we'd like to be in an ideal world, what, exactly, do you want to be done right now by members of the community?

It's not like we can all go in and change the Cloudflare code to stop using hCaptcha. The most we can all do is give alternatives and workarounds while pushing on Cloudflare and hCaptcha to support this scenario. Which is all being done in this thread already.

We can expect developers to think about accessibility during development. You would never see a building constructed today without a ramp or with insufficient handicapped parking. I realize that you can't change culture overnight either, but the fact is that technology like this, designed for use in large applications used by a huge portion of the population of the world, shouldn't need to be retrofitted to be accessible. It should have been baked in.

Yeah, and then... a few days later... The problem is reCaptcha, just stop using reCaptcha as Google is tracking all of us.

My personal opinion is that it's better to use hCaptcha. I wasn't aware that they don't support audio. I think a better approach would be requesting hCaptcha to implement the missing bit.

Giving less control to Google is better. Given the widespread tracking they have access to right now.

> Also this ask a friend solution is like telling someone in a wheelchair to ask a friend to help them up those steps instead of just installing a ramp.

I agree, and I could’ve been clearer. The Privacy Pass workaround shouldn’t be necessary and we should demand Cloudflare do better.

(To avoid any doubt: I’m not affiliated with Cloudflare in any way, other than that I’m a customer on their free plan.)

I think you need to take a deep breathe.

You're responding to a rando on the internet, not the CTO of Cloudflare. Your parent is just trying to offer help to OP, and you're mad because you assume he is a Cloudflare dev? Even if he was, it's not like he has unilateral control over product decisions.

> and just ignore social responsibility.

...and they're ignoring the law in many place.

Yes, there are limitations, and I'm sure hcaptcha will get there, but until then, a helpful person on the internet gave a helpful workaround.

Perhaps it's only a matter of time before volunteers or well meaning organization [1] add captcha solving to bemyeyes-style solutions.

[1] https://en.wikipedia.org/wiki/LightHouse_for_the_Blind_and_V...

There's AIRA and it can do it, if you don't care that another person will see what site you're trying to access and if you're willing to go through a super complicated techy procedure to get a free american phone number, which not everyone knows about. Also, not everyone speaks english.

There are a couple of bugs[0][1] in Privacy Pass that causes spending tokens during collecting, thereby preventing collecting tokens.

[0]: https://github.com/privacypass/challenge-bypass-extension/is...

[1]: https://github.com/privacypass/challenge-bypass-extension/is...

We also have a special accessibility mode https://www.hcaptcha.com/accessibility

Just like Google, it is also horrible for those who use more "nonstandard" configurations. I don't remember if it's CF or Google or someone else but I recall coming across sites that would block you if you have JS disabled, claiming that you're a bot, and the ban-page was the same across those sites, so it was a common "provider".

The "human detector" of the modern internet doesn't accept disabled people as sufficiently human. That's... the most dystopian thing I've heard in a while!

In the other direction, Google thinks I'm a bot if I use complex search queries (exclusions, site:, quotes, etc.) in an attempt to find what I'm looking for. Apparently it thinks I'm too intelligent to be a human.

You’re not profitable enough. You’re insisting on finding the thing you want, instead of settling for what Google is pushing today.

Since they're on HN they likely use uBlock, which means that they're not profitable for Google regardless of how they search

I get that you meant that sardonically but it's worth pointing out that it would be even more unprofitable implementing ways to catch and redirect you.

Seems like if this issue could easily be solved, a startup would have formed to solve it. Are there any alternatives to recaptcha and hcaptcha that are effective and also accessible?

Hi! I work at hCaptcha (we handle the CF captchas), and to respond specifically: audio is inaccessible for those with visual and auditory processing issues, as well as being broken by ML techniques. Not to project, but we suspect this is why Google turns it off if your browser looks at all suspicious.

Another user mentioned the Privacy Pass solution: https://news.ycombinator.com/item?id=23902870

While Privacy Pass is not perfect, we are working towards getting better. I'm not in the product flow for accessibility, but if you have ideas outside of audio that you believe would be able to distinguish between yourself and a robot, I'd be happy to discuss via email to ensure that we can get a solution for you and anyone who can reasonably interact with a computer.

eta: you can email me directly at work via: josiah@intuitionmachines.com

eta2: http://hcaptcha.com/accessibility

Audio captchas are considered broken completely (search for various papers over the past decade, including one from CMU), so it has limited usefulness only in a very narrow context in practice. I suspect this is a harbinger of things to come - as we close the gap on passing the Turing test, captchas are likely to get less effective, and we will need to transition to a very different solution for bot detection.

or, just let the bots be.

often it's easier to just use a website as api instead of using some broken xml nightmare that requires knowledge of the database tables.

If the concern is rate limiting then just rate limit the website. And if you don't like people operating websites with bots then I don't know, maybe stop making websites.

Easy proxying means rate limiting doesn’t really help all that much to defeat bots. And then if you do something like blocking or severely restricting something like Tor (the world's largest open proxy and hence, primary abusive traffic source; something which it would make sense to throw extra bot walls in front of), privacy and accessibility advocates jump down your throat.

This is a no-win situation. I’m not convinced it’s possible to have ones cake and eat it too, here. Someone upthread said “security, privacy, accessibility, pick any two”, and I have yet to see any evidence of a third option.

We offer privacy pass so that we don't force our users into the false dichotomy above. For our users it's a win/win situation

I think you do not understand why website owners use captcha. Website owners use captcha services, because doing so saves money for them.

Captcha will stop saving money, if the captcha becomes so ineffective that the short-term and the long-term downside (annoying users who are subjected to captcha) exceeds the cost saving ("cost" here is potentially many different things - it could be quality of service for normal users, it could be opportunity cost, it could be the cost of serving the traffic like network bandwidth or cpu or database capacity).

Yes there are subset of people with visual problems who also have auditory problems, that does not mean you should not support the ones with vision only problems.

Are there not "ML techniques" for visual captchas too ?

With more and more powerful models like GPT-3 coming along every few months, Even if the accuracy levels are less for visual over audio today how long do you think that is going to last ?

I believe they meant that audio captchas have become inaccessible to humans in general. Bots have gotten so good at audio captchas that the difficulty level has to be high enough that humans are also unable to complete them.

Visual captchas too can be quite very hard these days for the same reasons, there are many other ways to verify bot or human, many of them may not allow you to remain anonymous while doing so, pretty much any 2FA method could work for example.

I would say given the choice for a number of people the accessibility is more important than loss of privacy. At least it is choice they should have instead of shutting them off the internet.

Hi HCaptcha-er here. You don't lose all your privacy with our accessibility option. Also we always abide by DNT style protections even for accessibility. That being said our privacy pass solution is totally private and safe.

> Are there not "ML techniques" for visual captchas too?

There are.

> how long do you think that is going to last?

I don't know. So far it hasn't been difficult to recognize the garbage traffic when we take the time to look. And attackers always make it easier by releasing github source and / or announcing their results on Twitter.

Ugh. That accessibility login solution sounds awful, but at least it's available. I can definitely understand why you don't want to have an audio option, given that by all accounts they're next to useless.

Give it a try. It's so easy it feels like hacking.

Captcha submit requires two touches on iPhone 6S because the submit button will scroll out of view on first touch.

Do your devs or product people ever use your own product? That’s one of the first things I’d notice and fix.

I run android. Other devs run Android and recent iPhones. Someone may have an iPhone 6S to test and fix on, I'll check.

In the future, if you've got a bug that's preventing you from doing your thing, we have bug reporting instructions: https://www.hcaptcha.com/reporting-bugs

ETA: two different people have tested on Chrome and Safari on iPhone 6S locally, and can't reproduce. Please take a screenshot and provide more information to: https://www.hcaptcha.com/reporting-bugs if you want to get this fixed.

Is this on safari?

We test on iPhone 6s and it seems to work for us. Perhaps file a support ticket and we can take look?

It might be worth filing an ADA compliance notice against cloudflare. This is seriously disturbing.

As someone that works in ed tech - anyone that supports higher education is required to support screen readers etc. If any of those sites are using cloudflare that would block them from being compliant pretty seriously.

I'm not american, otherwise I would definitely do so.

This issue is mostly unnoticeable, as long as you're not considered malicious to Cloudflare, everything works perfectly and passes accessibility audits. When something weird happens and Cloudflare flags you, you cannot access the system at all.

For those doing accessibility audits out there, this kind of issues might be worth looking into, as they're very non obvious and very critical.

Have you tried our accessibility option? https://www.hcaptcha.com/accessibility

I'd love to read more about the economics of captchas - what is the "labor" of classifying images worth? What is the marginal cost per captcha of human solver farms, etc. Can anyone point me to some resources?

The a11y problem is super urgent for the majority of products out there. Not only it prevents blind/non hearing people of using systems. People also tend to forget that having a11y in place elevates UX on another level. It enables using apps/web with shortkeys, makes everything more readable, changes entire UX flows into simpler ones.

I have no affiliation with CloudFlare at all, but my interactions with them have all been kind, professional, and courteous.

I might recommend reaching out to the CEO, Matthew Prince, with a succinct explanation of your problem and I would be surprised if that by itself does not kick off some serious positive action!

As a founder of HCaptcha with a legally blind sister I take accessibility very seriously. We implemented this solution when we were still a tiny startup and I think our accessibility is by far the best of all the captchas. Please email support@hcaptcha.com if you are having any issues and we will find a solution. I'm sorry for the confusion and difficulty you may have suffered.

https://www.hcaptcha.com/accessibility

Honest question: I've never really understood the back of the napkin math of how Cloudflare functions economically, which I feel would go a long way towards my understanding of why/how they were able to become such an integral and generally positive part of the Internet.

Did they have some crazy in to get cheap bandwidth? Did they bet big on bandwidth prices falling? Did they figure something else out that nobody saw? Do they just to a tremendous job of migrating sites from free to paid plans?

Read our S-1, it's all in there.

https://www.sec.gov/Archives/edgar/data/1477333/000119312519...

II love cloudflare. It has really helped out with several sites/projects that I have worked on and the service is top notch. I am also an investor. I tend to invest in stuff which I use a lot or trust/respect the employees. Weirdly what made me really invest is the level of geekiness on the company. I remember seeing you guys using a lava lamp wall to generate entropy and just thought "that's awesome". I just wanted to say don't change. Your lz4 implementation, aes gcm golang optimizations have directly benefitted me. Coupled with really high quality post mortem articles, articles on interesting things like compression, encryption, networking, a few random articles (the privacy focussed file system recently comes to mind, written at the same time I was making my own distributed fs) just leave me with a lot of respect for the culture there.

I was worried a bit when I saw the initial IPO that the free tier would leave (despite promises it wouldn't) but that doesn't seem to be the case. Literally the only bad thing I've seen on the site is recently you switched from recaptcha to a new one that I had a real tough time with logging in today - it was a bit glitchy on my pc. The only suggestion I thought of as well would be a simple "maintenance mode" similar to the "I'm under attack mode" which would allow those of us without super-ha to quickly toggle on something to pop up a "sorry server"/site is down form maintenance page without having to mess with our proxies/web servers.

Anyway I know this comes across as totally kissing-ass but I just wanted to say thanks to someone who actually works there. Everyone fat fingers stuff every now and again,don't sweat it.

If you use Cloudflare workers, you could put together a maintenance page with a simple script + Cloudflare Workers KV as a toggle switch. It would cost $5/mo if you're not already using workers.

It's a bit of a sad story, but maybe you'll like this read about one of the guys who laid the foundation for their tech and his sad decline: https://www.wired.com/story/lee-holloway-devastating-decline...

I really liked the stories of his skill when he was in his prime. Very inspiring.

Sounds lame but this story really touched me. I recently visited a family member who's nearing the end and it broke my heart to see them in that condition.

I'm not sure what to take from this. But thank you for sharing it

Wow, thanks for linking to the article. What an awful story. Makes you think about the wisdom of undertaking major elective surgery if it's not absolutely required in the short term.

His wife saw it as a turning point, but the doctors and neurologists don’t seem to think FTD is linked to surgery in any way

I dont know how I missed the Cloudflare IPO and this Story. So thanks for posting it. Extremely good writing, though the story was very sad.

I wonder if there are any relationship between FTD and how clever a person or how much brain a person uses.

Nothing beyond the normal "cognitive reserve" findings that high education and broad cognitive engagement makes the brain pathology take longer to reduce your functioning to a dementia state.

Reminds me of this part of Robert Sapolsky's Human Behavioural Biology lecture series:

Robert Sapolsky - Phineas Gage and frontal damage https://www.youtube.com/watch?v=5wKDXzk8Wm4

Never heard of him but I read this article in one sitting (extremely hard thing for me) and I felt some unexplained connections with this story. A Beautiful Mind is another book which I very often go back to and go through similar emotions.

In all, it was a captivating read.

> I remember seeing you guys using a lava lamp wall to generate entropy and just thought "that's awesome".

To be sure (and for the sake of internet rando completenessism), it does look like CF waited until the original SGI patent on the technique ran out. :)

https://patents.google.com/patent/US5732138

> it does look like CF waited until the original SGI patent on the technique ran out.

As ever, innovation was brick-walled until IP got out of the way.

> To be sure (and for the sake of internet rando completenessism), it does look like CF waited until the original SGI patent on the technique ran out. :)

That's right. See https://news.ycombinator.com/item?id=23860658

I work at hCaptcha, we run CF's captcha. If you're having problems in the future, popping open a debugger and capturing the results can help us figure out what's going on. But also: browser, OS, site, ...?

Right now: there is an issue with Safari users on the most recent iOS and OS X, where 3rd party cookies have now been disabled by default. We're working on a solution.

If that's your issue, you can fix on your side in the short-term by not using Safari, or by enabling 3rd party cookies.

I'd love to see your reply to the blind person commenting on your product here.

Here it is: https://news.ycombinator.com/item?id=23903405

I cannot believe apple decided to roll that out in the middle of COVID. I got hit pretty hard with it and am finally rolling out a fix for my own stuff.

(posted in case others are hit by this) using `requestStorageAccess()` on a user click event seems to work and that's what we're rolling out :/ https://gist.github.com/iansltx/18caf551baaa60b79206

Congratulations with growing so big. Why aren't there dozens of lookalikes by now?

I mean CDNs that will let you override the origin's cache instructions and do a decent job of DDOS protection, and whose feature list otherwise looks a little like Cloudflare's.

Yes, why isn't the CDN a commodity?

CDNs are expensive to build, and often not very useful to customers until you've built out a large portion of it (actual hardware required, you can't just run it atop AWS). On top of this, much of the money is in Enterprise. So you've got to compete with Akamai, AWS, Fastly, Incapsula, Cloudflare, and several other notable ones to get any customers to speak of.

There are smaller CDNs out there. You can find them readily enough.

Yes, but for a customer it's trivial to swap out one CDN for another, isn't it?

Depending on how you integrated it and how deeply, it may be trivial. Certainly going from one DNS-based CDN to another can be pretty easy - a Cloudflare / CloudFront swap could be quick.

Which suggests to me that CDNs are already a commodity in some ways.

One answer - It's much, much harder to build and operate a global network infrastructure that it might seem. It's also even harder to invent some sort of "killer feature" or other genuine innovation on the experience. You're likely not using Cloudflare simply for commoditized pipes alone, but for other features or designed experiences in their offering.

A second answer - there are a bunch of bottom-barrel commoditized pipe services. You likely haven't heard of them because they're so generic. They've existed before Cloudflare, and more will be created in the future https://www.citrix.com/products/citrix-intelligent-traffic-m...

Hey John, thanks for taking the time to reply.

Respectfully, the info in the S1 (flywheels, etc.) seem to be what sustains you _now_.

Maybe a better question is "how did you identify and kick off that flywheel?"

Don't take this as being glib, but isn't this the entire point of funding rounds? Demonstrate growth and talk about a few approaches to creating a business model that you're investigating, which then brings investors on as they're betting you'll find one that'll succeed?

The relevant excerpt seems to be here:

Market Opportunity

We believe our platform disrupts several large and well-established IT markets. The key markets that are addressed by our platform include VPN, internal and external firewalls, web security (including web application firewalls and content filtering), distributed denial of service (DDoS) prevention, intrusion detection and prevention, application delivery controls, content delivery networks, domain name systems, advanced threat prevention (ATP), and wide area network (WAN) technology. From our analysis based on IDC data, $31.6 billion was spent on those products in 2018, which is expected to grow to $47.1 billion in 2022, representing a compound annual growth rate of 10.5%. We also are actively developing new products to address adjacent markets including compute, storage, 5G, and Internet of Things (IoT) that are not included in the estimate of our addressable market.

That’s just general market analysis. It doesn’t say anything about what their hedge is. Or was it simply that the existing actors were that inefficient/price gauging.

LOL, I'm sure it is, but referring someone that is wondering about the "back of the napkin math" to an approximately 250 page S-1 filing is sort of ludicrous.

Since you’re here, any response to the top comment re: inaccessibility to disabled persons?

I don't really know, but I guess the biggest feature in the past was protection against DoS attacks.

Bandwidth isn't expensive compared to what most people are paying for it. Cloudflare is paying for the infrastructure as it is to handle attacks, so why not use it? As long as it doesn't affect paying customers then it's great marketing.

> Did they figure something else out that nobody saw?

I may not have a full scope of the history, but my own experience with DDoS protection was quite different. Whilst providers offered anti ddos protection through GRE tunnels and dedicated machines behind DDoS appliances and a heavy null route hand, Cloudflare had a simple few-click solution that worked at the web application level making things a lot easier and, also, allowing for features like caching, and thus, CDN benefit from a global network. Further, they've maximized performance on their machines, and as a result, Cloudflare is wicked fast.

Cloudflare does what it does really well and has built additional services on their global network that make a lot of a sense and provide a lot of value.

Hats off to CF.

This has been my experience too, CF is just so easy to migrate to and to configure once you're there.

If AWS was able to offer a single click "DDOS protection and CDN" feature with similar pricing and features as Cloudflare then I'd consider it since most of our infrastructure is on AWS but at the moment they don't offer anything nearly as competitive. Just the Cloudfront bandwidth costs alone would dwarf our total infrastructure costs.

Indeed. I think I understand the argument claiming Cloudflare is a SPOF. However, if you're actually dealing with DDoS attacks with any kind of regularity, you're likely to have better uptime with them than without.

Cloudflare is a relatively cheap "lite CDN" for developers who want caching without putting in any work. This becomes a gateway drug for Cloudflare's more expensive plans once you outgrow the free plan. It quickly adds up; I worked for a company that wasn't even on an enterprise plan and was spending hundreds of dollars a month on Cloudflare just because they had a lot of domains.

My reservation with Cloudflare is the concept of letting a third party MitM my SSL traffic. That and it's more expensive than a cheapo CDN like Stackpath if all you really care about is CDN (and Cloudflare isn't even really a good CDN, just a quick hack to speed up small static files).

In my experience, it's extremely rare to find a CDN that doesn't expect to do TLS termination. My understanding of TLS is that it's exceptionally difficult to cache content if you cannot see into the requests.

Perhaps I have overlooked something?

Right, with Stackpath they do TLS but not necessarily on the primary domain. You don't have to point your nameservers if you don't want to. So you can set them up on a subdomain and use their fake SSL to serve purely static files (if the file doesn't exist on CDN at the time it's requested then their system will pull it directly from your server via a private subdomain, serve it to the client and store it for next time)

So in this way it's possible to setup CDN with shared SSL for purely static files but not the app server itself; you don't have to give the keys to the whole kingdom so to speak and it's cheaper than Cloudflare at the basic level.

Let me see if I follow. Auto-provisioned TLS (misleadingly termed "fake SSL") on the front-end for delivering static contents and caching. A private subdomain with a pinned cert not managed by the CDN to deliver static contents to the CDN. And a third subdomain for the application itself that's not going through the CDN.

I was under the impression that the same result could be achieved with Cloudflare, or indeed nearly any CDN. Was I mistaken? Though you may not actually need a secret, private subdomain for static files with all CDNs.

Again, please let me know if I've made a mistake somewhere. I'd love to learn something this morning.

You're right about that. So it might look like this

static.domain.com (CDN subdomain with auto provisioned TLS)

static-uncached.domain.com (private pass-through subdomain when CDN is missing a file)

www.domain.com (app server hosted wherever)

You're right that you could do something similar with other CDNs including Cloudflare (you can just set the www subdomain to "bypass Cloudflare" to accomplish a similar result), but I'm not aware of any way to use Cloudflare on a domain without forwarding your nameservers to them, effectively giving them complete control over the domain. At least with Stackpath I can host DNS wherever and simply point the subdomains I want at them.

Also, by the time you do the work to split static files into separate subdomains you might as well go with a dedicated CDN. One of the selling points of Cloudflare is for sites serving everything on one subdomain that they can forward to Cloudflare and get caching without any work.

They have a CNAME setup where authoritative DNS stays outside of Cloudflare: https://support.cloudflare.com/hc/en-us/articles/36002061511...

It requires at least the Business level plan, though.

Ah my bad. That’s probably why I never knew about it. The $200/month entry price is steep there.

> My understanding of TLS is that it's exceptionally difficult to cache content if you cannot see into the requests.

Not even "exceptionally difficult", but flat-out impossible. From the perspective of an observer, TLS sessions are random data. The protocol is specifically designed to defeat attempts to replay data -- a CDN is indistinguishable from an attacker in that sense.

Hundreds of dollars a month doesn't seem material for a company.

Nope, but I'm sure lots of companies are in this situation, where they just barely need the features of the lowest paid plans, end up scaling across several domains and Cloudflare makes a ton of money off a vast majority of customers that will never need their more advanced features.

Plus that means hundreds of unique domains with unique content. If you have hundreds of domains all pointed back to one site with a single forwarding rule, 99/100 could be free tier accounts.

> My reservation with Cloudflare is the concept of letting a third party MitM my SSL traffic.

I wonder what the Venn diagram of people who insist every website must use HTTPS for privacy reasons and Cloudflare users looks like.

Unless you're sending all your traffic back to physical machines that you own locked into a cage in a datacenter, you are probably letting someone MITM your SSL traffic. For example if you are hosting on AWS, Amazon has access to your keys. If you are hosting on a hardware server leased from Hetzner, Hetzner has access to your keys.

When a 3rd party has access to your keys, their responsibilities to you are spelled out in your contract with them. That's true for CDNs as well as hosting companies.

It's more complicated.

For most websites today if someone can intercept traffic somewhere close to the server they don't even need the keys, they can just fake responses to pass CA validation and issue valid certificates with their own keys and MITM like there is no encryption.

And coldboot attacks performed by a hosting provider staff of dumping memory and finding keys isn't that realistic of a threat, just like putting servers into a locked cage on someone else's property isn't much of a protection.

I send traffic for my sites and apps to physical machines that I own and operate in a secure location, but I doubt most people are doing this.

There’s a difference between a VM host with the technical ability to carry out a targeted MITM attack against its customers using hardware-level access, and a provider that sells MITM as a service.

You don't need to use your SSL certificate for the CDN. Most websites that use CDNs only do so for static assets such as images, fonts, CSS, etc... the dynamic content comes from the origin server, encrypted with a private key that is not handed out.

If you press F12 in your browser and navigate to some major sites, you'll notice anywhere between 1-5 different CDN domain names that aren't directly related to the original host in any way.

As far as I understand it it's a freemium B2B model: If you're small you probably can get away using the free tier. Then when you get bigger you outgrow the free plan, either because you want specific features or because your bandwidth becomes too high.

That said bandwidth really isn't that expensive, at least if you're buying it at the scale that Cloudflare does. Many people seem to be used to the bandwidth prices of the large cloud hosters, which are really insane and have been marked up by a large multiplier to disincentivize people to transfer their data elsewhere for processing.

The NSA/CIA have a huge hidden budget. They are known to invest in startups. Why try to break crypto when you can just pay some tax payer money to large corps to get them to convince sites to mitm themselves.

Worth noting that cloudflare does a poor job of identifying and allowing crawlers specially Google. I have a site with 1 mil URLs , nothing big and every time you turn on cloudflare it reduces the crawler to 1k visits or less per day. If you compare your logs for visits for Google and Bing with CF turned on/off there is a huge difference.

Suffice to say, that in over 5-6 projects I have added CF to, it's never worked out in my favour.

> I have a site with 1 mil URLs , nothing big

A million URLs isn't big?

I wonder why. May be CF should work with Google and allow those crawlers?

Do I understand correctly that you're suggesting Cloudflare reduces organic search traffic due to restricting search engine crawlers?

Interesting perspective, but it seems like this is just an ad for easyDNS and their "Proactive Nameservers," though I couldn't imagine a better time than the misstep of a behemoth of a competitor in this space. Not to detract from the more important discussion about the internet's dependence on Cloudflare overall.

Hey, never let a competitors misfortune (misstep?) go to waste!

It may be just an ad for easyDNS' Proactive Nameservers [1] product but it provides a roadmap for one possible solution to this type of problem. From a quick reading of the marketing info, the solution can be summarized as "Provision, Monitor, and Fail-Over DNS Name Servers across multiple DNS-as-a-Service providers". The question I have is whether the following constraint is artificially introduced or not:

> We must be your domain registrar for this to work...

IIRC, Netflix OSS published some tools quite some time ago to support multiple DNS providers but I don't know/remember if they tackled the availability problem. The question comes down to build vs buy and whether the solution is general enough to warrant an Open Source Software solution.

[1] https://easydns.com/dns/proactive-nameservers/

From TFA:

> The only requirement to use Proactive Nameservers is that we have to be your registrar, because we need to connect to the registry to update your nameserver delegation.

So I guess technically this could be achieved with an API for your domain settings.

> IIRC, Netflix OSS published some tools quite some time ago to support multiple DNS providers but I don't know/remember if they tackled the availability problem.

The classic way of doing this is AXFR (your own DNS server is a "hidden master" and the DNS providers are the slaves).

The problem is you won't be able to have redundancy at the registrar level, but that has historically at least been less of an issue.

They want to be the registrar to be able to update your NS records. But ... that's not really important nor needed (So the answer to your question yes, it's likely artificial). Just use two anycast-ed IPs/domains. (Like Cloudflare.)

The magic happens at BGP level.

I considered CF as a domain registrar, but they don't allow setting the NS records. So you must use them. (They basically use sane no-nonsense domain registration as a way to gain leads for their main product. Pretty smart actually, because it's a great high-level add-on for their main product, but they just went ahead and made that the bait for everyone.)

Anyway, ideally, if you add 2 separate sets of NS servers to your NS records then you eliminated this SPoF, great. Sure, it's your job to keep them updated, and in sync (preferably, to avoid problems like half of your users landing on a different CNAME/IP/etc).

And recursive nameservers will handle the failover.

> Just use two anycast-ed IPs/domains

What are the brands offering DNS at a flat rate over anycasted IP over a few continents?

Most of the offers I see are per query at high rates. Setting up my own ASN to do this would be too expansive in IPv4

I meant that easyDNS should handle the BGP for its clients, without requiring their clients to use them as registrars.

There's HE.net's free DNS, and though they don't explicitly advertise as, it's anycasted. (Check via https://tools.keycdn.com/ping , try 216.66.80.18 [ns5.he.net].)

https://www.cloudns.net/premium/ seems to be quite affordable with no query limits :o

easyDNS has to be the registrar because only your registrar can change your nameserver delegation with the registry. This is, in essence, the registrar's job. To maintain your domain record and info, including nameserver delegation, with the registry.

You could do it with BGP, but it is non-trivial and you need your own ASN to do that.

But you can just add multiple DNS providers yourself. I mean you can add the namservers of both easyDNS and cloudflare. EasyDNS just automates this.

In theory they could simply create a few subsidiaries, let's call them saferDNS1,2,3 and have them build completely different redundant DNS architectures, and add then add the resulting nameservers.

That said, it'd be good to see an actual domain that uses this "proactive" feature to see what easyDNS is doing.

I'm not a DNS expert, so... It's not really that simple is it? If you have multiple nameservers I thought they get equal weight, don't they?

So if you have Cloudflare + (ex:) NS1, and you're using Cloudflare for caching, you need your NS1 records to return Cloudflare proxied IPs normally, but origin IPs under failure conditions. That's a lot of infrastructure.

It also fails completely if you're relying on Cloudflare for DDoS protection and IP obfuscation because a failure means your origin IPs get exposed. That's assuming Cloudflare DNS being down means Cloudflare proxying is down too. It might not be the case, but I think you'd have to plan for it.

Then there's also Cloudflare's detection of nameservers. I haven't tried it with more than Cloudflare's nameservers set for a domain, but if your domain doesn't actively use their nameserver they'll drop your site from their system. So, at the very least, you can't use Cloudflare as a secondary DNS provider (at least the last time I checked).

I was talking about just the DNS layer, but ... you can periodically check what IP Cloudflare would return and return that. (There's also the apex-CNAME record type, ALIAS or DNAME, I don't remember right now, but PowerDNS supports it, and you can set up the resolver to use CF's NS.)

Of course CF doesn't support anything like this, but it works well (because they use quasi fixed, static anycasted IPs for the HTTP(S and TCP?) proxying/load-balancing too), even if it's hacky as hell.

If they have any active verification of nameservers, and if they disable the proxies if they detect something bad, then ... it won't work obviously :)

But technically there's nothing amazing in being a registrar of a domain. So both CF and easyDNS are just stubborn in the name of user experience (consistency).

... all in all, working around any SPoF (reliably) will usually require exponentially more resources/engineering/care.

> Proactive Nameservers is a patent-pending system that optimizes the nameserver delegation for your mission critical domain names.

That's a huge negative and I can't believe they think it's a good marketing point. I don't want a patent encumbered, non-standard solution for critical infrastructure.

> We must be your domain registrar for this to work.

So they're updating the domain record at the registry level to facilitate failover? That's the only scenario I can think of where they _need_ to be your registrar. Assuming that's the case...

I've always seen 24-48 hours quoted as the worst case wait when updating nameservers at the registry. I've never seen an explanation of how it works, what's allowed to be cached, how long it actually takes to update, etc.. How do they do it in a way that's suitable for failover? Do they have a special SLA with registries?

How would the registries handle a deluge of nameserver updates? Imagine a Cloudflare scale failure and corresponding registry updates. Would the registry servers be able to handle it?

I'd love to see a technical explanation of how their proactive nameservers system works.

I think people are Cloudflare fans simply because so many other services suck more.

I gave Cloudflare a fair shake. But after hearing their lies way too many times, I'm calling them out for being deceptive and unscrupulous.

After being told that sites pretending to host Adobe Flash updaters and pretending to be Bank of America can't be taken down by Cloudflare because of their rights to free speech, I knew their attempts to pretend to be one of us, attempts to pretend to care, were nothing but bullshit.

They claim they don't host. If hosting DNS is not hosting, then what do you call DNS hosting? You literally CANNOT use their abuse web form to report domains which use Cloudflare just for hosting DNS. They do not handle abuse sent to their abuse email address (they simply send a form response saying to spend ten minutes filling out their crappy form that has all sorts of problems).

Of course, their web proxy services are also "not hosting", even though they're protecting all sorts of scammers.

So why should we think they're not bullshitting us when they run 1.1.1.1 and tell us they're not logging? Why should we trust them more than our ISPs by running DoH through them?

They WANT us to be dependent on them, because the more control they have, the more money they can make. It's dangerous, and they've shown they have no honor.

I've genuinely tried to correspond with them on Twitter, and they excel at not answering the question asked but instead just diverting. It's scummy, unprofessional behavior and I encourage everyone to consider whether they deserve anyone's data or business.

> They WANT us to be dependent on them

I think this is an important point about CloudFlare that can't be made often enough. It's been some years since I first noticed, and it seems as true today as it was then: wedging themselves into core Internet services and data flows seems to be an intentional part of CloudFlare's strategy.

There is no case given the architecture of the Internet where one company need be exposed to so many traffic flows from millions of people. The search engines got there first and we eventually stopped complaining, but this does not make it any more justifiable to copy the model.

What reason would a company have for desiring this outcome? We know Google can detect and predict flu outbreaks. Imagine what is possible when you have every click on every target web site.

There is a fair chance their data is already approaching the comprehensiveness of Google, and I'd be surprised, if not disappointed to learn they were not already working on unannounced (now or eternally) internal intelligence products based on that data. There are simply too many pockets who would be willing to pay for it.

I thought 'we' were for 'Net Neutrality'?

Remember, carriers and service providers not allowed to decided what to do based on content?

Yeah, Cloudflare shouldn't be censoring anyone. I could understand if it were requested by law enforcement but why would we want them trying to police the web?

Damned if you do, damned if you don't.

Seeking problems is another way to write it.

refusing to provide your services to someone is not censoring them. did they black hole their dns entries in 1.1.1.1? did they steal their domain name? no. companies fire customers all of the time.

For me, the biggest disappointment was when they were defending hosting terrorist site for "free speech" reasons:

https://www.fastcompany.com/90312063/how-cloudflare-straddle...

https://blog.cloudflare.com/cloudflare-and-free-speech/

> the company serves at least seven groups on the U.S. State Department’s list of foreign terrorist organizations, including al-Shabab, the Popular Front for the Liberation of Palestine (PFLP), al-Quds Brigades, the Kurdistan Workers’ Party, al-Aqsa Martyrs Brigade, and Hamas.

> CEP has sent letters to Cloudflare since February 13, 2017, warning about clients on the service, including Hamas, the Taliban, the PFLP, and the Nordic Resistance Movement. The latest letter, from February 15, 2019, warns of what CEP identified as three pro-ISIS propaganda websites.

All this while routinely censoring other sites. Their actions are very different than their words.

From the article:

“ This question assumes the answer. A website is speech. It is not a bomb. There is no imminent danger it creates and no provider has an affirmative obligation to monitor and make determinations about the theoretically harmful nature of speech a site may contain.”

Are DDOS-for-hire sites "speech"?

Their only use is to suppress speech.

Well that would depend on what the target of the DDOS is wouldn’t it? Either way the point being we want companies out of politics. They have no business in them.

That is the problem with massive centralization even if it is market level and internally Cloudflare (or any other big fish) does decentralization/fail-over of their own. Many of these companies should have had fail-over to competitors at least for reliability.

The problem with near market monopolization, oligopoly, even the singularity, the fail-case is catastrophic and may even wipe out decentralized, diffused, dispersed, decoupled system solutions that can't make it due to so much relative size from the big fish that it squashes them along the way. The bigger the ship the longer it takes to turn.

This Cloudflare issue is like the recent Facebook SDK startup crashes where everyone has a single point of failure on Facebook SDK where people should be using or able to use the OpenGraph API directly as they need which is more robust to the app that uses it, it won't crash on startup.

In business it is a goal to centralize to grow, in nature and robust systems it is more differentiation and decentralization to survive. There will always be a push and pull between these two forces.

Systems and markets are like gardens. The garden must be maintained, new seeds planted and helped to grow from small to mid-sized, mid-sized plants the bulk of the garden, and then the larger plants need to be culled back when they get too big to not take the mid-sized and then all the resources from the new seeds/small plants. The problem is we have allowed the top end to take over the garden and when they fail they fail spectacularly. The bigger the scale the bigger they can fail.

> Many of these companies should have had fail-over to competitors at least for reliability.

How would you even set such a thing up? I fear that you might get a couple of collusionary companies that bail each other out and smaller providers might just be left out to dry…

Many DNS providers have the ability to do AXFR "zone transfers", so you can sync your records to a secondary provider and you would add a secondary set of nameservers for redundancy. Unfortunately Cloudflare doesn't offer this unless you pay for their Enterprise plan (they started offering it earlier this year).

I do love using CloudFlare for DNS, lots of great features and generally works well, but I wish they would support AXFR for the lower tiers. I've been working on a solution for this using the CloudFlare API, but we'll see how well it works out.

I think they want to price discriminate: the costs from AXFR should be minimal, as it is an old technology, very optimized and low bandwidth.

However, cloudflare decision turns cloudflare "free" offering into a free SPOF. They should extend this offer to their free users, who could then use the secondary DNS that most hosts/domain name sellers provide for free to the IP that is proxified by cloudflare. It could even be limited to the case of proxy or cloudflare DNS failure, so that cloudflare could still price discriminate (make AXFR fail, unless cloudflare is down, like a dead man's switch)

Amazing that this isn't "free" and folks have to resort to syncing records across DNS providers with proprietary, vendor-specific APIs. AXFR is standard. It's not just Cloudflare... AWS and Azure don't support it either.

With Cloudflare's popularity, I'll bet if they started supporting it, others would too.

Fwiw, you could use stack overflow DNS Control to manage your DNS records and upload them to many providers. Then the only thing you'd have to do is flip a line of code to fallback to the other DNS provider.

Nice, didn't know about that tool. Checking it out now: https://github.com/StackExchange/dnscontrol

Back in the day (I haven't run my own mail server for years and years) there was a company called Secondary MX.

That's all they did. They weren't a mail host or provider, there was no UI, nothing. All they did was allow you to specify them as, well, a secondary MX, so if you were offline they'd cache your inbound email until you were back. Simple. Efficient.

The article posted actually talks about EasyDNS's implementation of exactly this.

> At easyDNS we experienced so much pain from this reality that we created a system to automate flipping DNS providers at the first sign of trouble.

> We call it Proactive Nameservers, and we’re the only company in the world doing it for some reason. Maybe this is because in order to provide a service like nameserver failover, it means a company has to admit to its customers the reality that their own nameservers may at some point, fail.

I imagine the implementation could look similar to how cell phones can use other carriers networks (for 911) when they don't have signal from their own carrier.

Most companies refuse to handle the accounts/sites cloudflare handles.

The bigger problem is that technology advancements gravitate at the demands of the noisiest, with the most social gravity.

Yes there are a lot of smart people at FAANG and Cloudflare corp.

There are a lot of just as capable folks not driven by job addicted meme.

Technically there’s no reason the web couldn’t be replaced with 1:many via Wireshark key sharing based access control to local content.

But via Wall Street, along these very particular rules, is how we are told to trade information. How is that not a planned economy?

Not just by doing what we’re clearly interested in doing naturally.

Make no mistake: big corp isn’t making us login at gunpoint. “They” didn’t do this. “We” did this.

Could you please stop creating accounts for every few comments you post? We ban accounts that do that. This is in the site guidelines: https://news.ycombinator.com/newsguidelines.html.

You needn't use your real name, of course, but for HN to be a community, users need some identity for other users to relate to. Otherwise we may as well have no usernames and no community, and that would be a different kind of forum. https://hn.algolia.com/?sort=byDate&dateRange=all&type=comme...

The question is: Can we do better? A natural monopoly is a good thing. It just means that the natural monopoly needs to build its own redundancy. Cloudflare total failure isn't common.

I wouldn't classify a natural monopoly as a good thing. It mostly signifies high barriers to entry for competitors and a market that's more susceptible to failure, as seen here.

I guess but there's not really a shortage of commercial CDNs. A market with lots of competitors but a clear winner says more about consumer preference and network (ha!) effects than it does about Cloudflare's moat.

> A natural monopoly is a good thing.

How so?

If you have a "monopoly" which is this case just means you're the market leader by a wide margin, but in theory this could mean near total control of a market but the reason you have this total control is because your customers choose to buy from you over your competitors simply because you're better and or cheaper then this is a good thing. Customers are getting what they want. And the market leader in this situation will have a hard time abusing their position because any they'll have competitors nipping at their heels if they slip up.

I would personally phrase it pedantically as not that the natural monopoly is good but that it exists because they are good. A fine distinction and a baked in assumption admittedly that if they were to no longer be good they would no longer dominate. Technically there are some other variables like if an extended stay on top would atrophy any competitors or not and the time scale operated upon.

On a century scale individual murderers aren't a huge concern to society because they are either dead or infirm by then.

> A natural monopoly is a good thing

I think for the sake of a diverse community and economy, monopolies should be difficult to achieve even naturally.

But for this specific example, a robust technical redundancy doesn't stop CloudFlare from going out of business. A technology company going out of business is pretty much the norm. Incumbents are a relatively new phenomenon for technology (sans some key exceptions), and I don't think CloudFlare is an incumbent. They are an accessory, and your business would probably run without them.

Cloudflare isn't a monopoly. Cloudflare doesn't even have a particularly strong moat when compared to Fastly or Akamai. Cloudflare doesn't even have any network effects.

I must disagree, and wish I had more data on how many sites host with cloudflare because it's free. I just checked, fastly is $50 per month, and Akamai doesn't list pricing so let's assume it's more.

Cloudflare has had network affects from integration with wordpress and cpanel too I believe for some time now.

without cloudflare your site can be taken offline by any random person willing to spend $20 for ddos sellers.

The free plan is a pretty big moat imho, especially if 'half the internet can fail..' - I doubt 10% of those sites would be using cloudflare if there was a monthly fee pushed on them.

Admittedly, my cloudflare usage is only about 30 sites, so my data point is small. The few hosting and design clients I have and their budget constraints are not indicative of 'half the internet' - but I don't believe fortune 500 and SV sites are either.

There is no CDN monopoly. There are several to choose from. Cloudflare is one of the new kids on the block.

There is no cloud monopoly either. Customers can choose from AWS, GCP, Azure, and several others.

The problem is not market concentration. There are plenty of options. The problem is customers choosing to put all their eggs in one basket.

How do you do a fallback from cloudflare failing when they’re your dns provider too? Any redirection around would take too long to change wouldn’t it? They’d be back up and running before it was implemented. What’s the right approach?

Don't use them for DNS, just point 60 second TTL A records at them.

I don't think cloudflare supports such a configuration.

The DNS is part of the load balancing, they serve different IPs based on location of the DNS query.

Edit: Apparently they do support a CNAME configuration if you pay for one of their business plans. That gives you the option to quickly switch away (if your TTL is low enough) but will impact performance by having to fetch the CNAME every 60 seconds.

Does cloudflare actually do geo-loadbalancing via DNS A records now? For years they only did anycast, unlike, say, Akamai, which hands out different IPs for each POP.

Actually, I'm not sure if they do any DNS geo-loadbalancing. I've seen it report different IPs from different locations at times, but that could be something else.

But I'm pretty sure they use DNS do other loadbalancing and DDoS mitigations.

For example, if a site is under attack, they can send it to different IP addresses to keep it away from other sites. Or if someone is directly targeting a cloudlflare IP with a DDoS, they can redirect all sites to other IPs and just blackhole that IP.

Where do you put your DNS servers? On-prem is going to be less reliable than cloudflare which has a 100% uptime SLA. I doubt your local ops team can compete.

my thoughts exactly. still none the wiser what the solution is.

For starters, couldn't you have name servers at multiple "places" for your domain? Name servers 1 at Amazon and nameserver 2 at Azure, for example.

Shouldn't be too hard to build an abstraction layer that updates them all when you need to make changes.

True, however at AWS at least, customers are specifically told "multi-cloud doesn't allow you to fully leverage the benefits of AWS", whatever that means.

It makes sense that cloud companies are inclined to keep customers from giving money to competitors, but they way they sell it and structure services, reserved instances, and enterprise discounts is such that you basically are putting all of your eggs in one basket.

People are given the agency to make their own decisions.

Nobody got fired for buying IBM, until they did.

Probably means that if you use a service that only aws offers and you failover to a competitor your product/service/website will be degraded

> "multi-cloud doesn't allow you to fully leverage the benefits of AWS", whatever that means.

One of the selling points of cloud providers is managed services like SQS. If you run a multi-cloud architecture, you either can't use managed services, or have to build abstraction layers on top of them (and only use the features that exist in both cloud providers' versions of the managed service).

If you want to use a managed service that only exists on AWS, then that's obviously incompatible with a fully multi-cloud architecture.

> If you run a multi-cloud architecture, you either can't use managed services, or have to build abstraction layers on top of them (and only use the features that exist in both cloud providers' versions of the managed service).

And this is, of course, why they do everything they can do discourage it. Because if you do that, not only are you not reliant on them for availability, you can switch more of your business to the other provider(s) based on current pricing, and they do not want that big time.

I thought that was something trivially obvious stated about fully leveraging the benefits - that they don't control other's clouds and thus they can't use all of the same sorts of performance or efficency boosting tricks. (People would be way more mad if they did as it would require hacking into say Azure to gain root access.) Not a domain expert but it is my interpretation.

If you make the engineering decision to go multicloud for whatever reason those are inherent trade offs you need to be aware of. They have their own agenda of course in addition to any actual fundamental "real" in bulk efficiencies that price reflects.

The problem is customers choosing to put all their eggs in one basket.

Until relatively recently absolutely none, and now almost none of the tooling allows effective multi-cloud or hybrid cloud/private.

Basically the cloud providers work very hard to prevent the commodification of their services with special incompatible service offerings, lock-in, interdependency, deep and opaque APIs for integration, and networks of training and certification that position change as a direct threat to people's job security.

Cloud providers today are basically Microsoft in the 90s.

Much as open source challenged Microsoft, I would say that the world now needs open infrastructure tooling that positions hybrid and multi-cloud as first class infrastructure architectural cases in order to displace established cloud provider hegemony. Even then we will have to fight the hardware and real estate economies of scale available to large established cloud providers.

I wrote some observations about this space based significantly on HN community comments prior to the rise of Docker a few years ago: http://stani.sh/walter/pfcts/ ... click 'original' ... the conclusions still seem timely.

> Until relatively recently absolutely none, and now almost none of the tooling allows effective multi-cloud or hybrid cloud/private.

Until relatively recently, the cloud didn't exist.

EC2 was launched in 2006. The notes I linked to date from 2014.

The cloud is marketing speak for what has been going on for decades. In the 70's, it was called time sharing.

> Until relatively recently, the cloud didn't exist.

Depends:

> IBM and other mainframe providers conducted this kind of business in the following two decades [after 1961], often referred to as time-sharing, offering computing power and database storage to banks and other large organizations from their worldwide data centers. To facilitate this business model, mainframe operating systems evolved to include process control facilities, security, and user metering.

[…]

> In 1998, HP set up the Utility Computing Division in Mountain View, CA, assigning former Bell Labs computer scientists to begin work on a computing power plant, incorporating multiple utilities to form a software stack. Services such as "IP billing-on-tap" were marketed. HP introduced the Utility Data Center in 2001. Sun announced the Sun Cloud service to consumers in 2000.

[…]

> In spring 2006 3tera announced its AppLogic service and later that summer Amazon launched Amazon EC2 (Elastic Compute Cloud).

* https://en.wikipedia.org/wiki/Utility_computing

Cloudflare has done really well in a short amount of time, but I'm sure Akamai is way bigger. Cloudflare is best known because they offer free/cheap tools for individuals and this post is targeted at developers. Most big enterprises I've worked with are on Akamai.

I enjoyed the analogy to gardening. Made me think of the terrific principles portrayed in the book The timeless way of building.

What's funny about this outage is I'm sure many of of us (myself included) used this window to analyze large services and determine an increase in major Cloudflare customers and presumably, revenue. Even ISPs like T-Mobile faced issues due to the Cloudflare outage! The situation has exposed just how critical Cloudflare is.

I went ahead and bought calls ahead of NET earnings next month. Cloudflare is becoming an increasingly bigger part of the internet backbone. Purely speculating here, but I wouldn't put it past AWS or another large player acquiring them soon.

> I wouldn't put it past AWS or another large player acquiring them soon.

I really hope it doesn't come to this sadly. I'd be okay with DO or somebody who isn't as massive doing a merger. Maybe they can pull resources to make each other even successful and maintaining reasonable independence.

> an increase in major Cloudflare customers and presumably, revenue. Even ISPs like T-Mobile faced issues due to the Cloudflare outage!

Careful about this methodology. Some services at my org were impacted despite not being direct CloudFlare customers. They had external dependencies that used CloudFlare.

So it's much bigger proverbial 'blast-radius' lest something happen to CloudFlare? Can you elaborate a bit on that part? I'm interested in knowing more.

Simple case of dependencies failing. Not much to elaborate.

e.g. NPM.js uses CloudFlare DNS, so services which needed to talk to NPM.js weren't able to do so.

> Purely speculating here, but I wouldn't put it past AWS or another large player acquiring them soon.

Internet™ by Amazon.

I really hope it doesn't come to this. I assume such a move would create a vacuum for a competitor. Not everyone wants to be completely owned by AWS.

“Cloudflare apparently fat-fingered a routing update and sent all of their global traffic to a single POP, vaporizing it almost instantly.”

Made me chuckle, as it gave me the image of a large server in some massive server farm glowing red, then bursting in a massive burst of light as dozens of bearded Sysadmins run out of the building screaming.

In my experience the sysadmins would be running in the opposite direction.

As usual, a relevant one: https://xkcd.com/705/

Didn't people also say the same thing about AWS a while back when that had a downtime? I guess the internet has multiple "Single-Point-Of-Failure"s.

Ironic since the internet (ARPANET) was specifically designed to not have a single point of failure

Doubly ironic since in doing this, they created a system where a protocol is the SPoF; namely nonsensical or false BGP advertisements can quickly kill the internet as a whole if done correctly.

That's not actually a contradiction. When an individual website is considered as a system, it will have multiple points where the failure of a component inhibits the system. It's possible to have both cloudflare and your database as "SPoFs" and that "single" is not meant to imply everyone only gets one.

It's absolutely true that if us-east-1 in AWS has a bad day, a significant fraction of the American digital economy will shut down. For some companies, the same is true of Azure and Google's various comparable offerings.

I read your post as skeptical. Why would you be skeptical? If you care about keeping your product up, you absolutely should have a fallback for cloudflare if you're a customer of theirs. Now, you might not care (and actually, for most folks I submit you need not care), but the folks making sure Ambulances get timely push notifications and realtime driving instructions probably care quite a bit.

A "single point of failure" doesn't mean there's only one of them. It means the whole fails when the SPOF fails. But you can have many of them.

A steel chain made of links has as many SPOF as links.

it's a series of single points of failure on different levels, so, not multiple points of faiure, but much worse, single point after single point, which means the house of cards fails as often as any of them. The internet of 2020 is a monolith

This is why AWS is puts effort into blast radius reduction.

The Physalia paper is a wonderful read that explains both the justification and high-level approach to implementing said isolation [1].

Another great example is shuffle sharding [2], notably used in Route53.

The end goal of this is that aws outages would be isolated to subsets of aws customers where possible. So instead of half the internet failing it's only 10% (obviously the exact numbers depend heavily on implementation).

[1] https://assets.amazon.science/c4/11/de2606884b63bf4d95190a3c... [2] https://aws.amazon.com/builders-library/workload-isolation-u...

Many of us don't even know it. My DNS for PortableApps.com is run through Digital Ocean, which uses Cloudflare for DNS.

On an unrelated note. Thank you so much for PortableApps.com! It was invaluable at university when studying in the library. To this day, I still use it for utilities that don't need to be installed on my desktop.

I am also grateful for PortableApps.com. Along with Scoop and Homebrew, they make using a system without root or admin privileges a really nice experience, in both Windows and Linux.

My sincere thanks to John and all the PortableApps.com contributors.

You're welcome! I'm glad it's helping you be more productive!

You're welcome, I'm glad it's helped and helping you! I try to keep it growing and relevant with a more synced cloud folder/work or school laptop/keep work and personal separate bent these days.

Okay here's the thing. I'm okay with people bashing other (competing) companies when they do wrong. However, I believe it is somewhat childish and uncalled for to bash another company, because of a mistake.

First of all "I use easyDNS so I didn't notice it at all tbh" is not only a childish assertion, it's borderline a falsehood. You DO NOT offer the same services, nor the same scale. (No, VOD would not work if your VOD provider used Cloudflare's offering.)

Second of all, as some have noted in other comments, you are very welcome to get just as big as them if you can offer similar (excellent) service and similar extremely competitive pricing. Otherwise, keep working on your offer and stop going for low hanging fruit like bashing the competitor for an outage when they literally might handle 1000x your traffic, and perhaps offer 20x the services your offer.

Just a little rant ...

Honestly I didn't notice the domain name, and I actually thought the author was being quite understanding by saying things like "This is inevitable and unavoidable and entirely excusable. Everybody blows up, every DNS provider in existence will experience downtime. No exceptions." and that they use Cloudflare themselves.

This is obviously subjective, but to me it didn't come across as "they suck use us" but rather pointing out the inherent flaws in this quite popular SPOF and cautioning to avoid it.

the tone may have been moderate, but the message is smug.

otoh it’s just marketing, and CF is no stranger to it, so i think it’s fair.

I think the root cause (which, IMO, you correctly point out) is lost on many modern developers.

For whatever reason there's this modern idea that if a company A is paying money to company B for a service, that company B will handle all the 'hard stuff' for them.

The end result is we have a lot of applications/infra built with SPOFs, in some cases known, but in many, swept under the rug and abstracted away to passing the buck in case of a large failure (i.e. major AWS/Cloudflare/Azure outages).

You also see this at times when vendors pitch internal software solutions. I've been at more than one shop where a vendor's 'silver bullet' turned into a SPOF time-bomb because nobody considered this company's solution could fail. After all, the sales presentation said it had %nines%!

"Your app will go down when half the Internet goes down" is not that big of a deal to most software companies, because:

1. no one's going to blame me if my app goes down when half the Internet is also down but they are going to blame me if my custom solution to the same thing causes an outage,

2. there's no way my custom solution is going to achieve the same uptime. AWS/Cloudflare/Azure are not perfect, but whatever I roll for myself is almost certainly going to be much less perfect.

Regarding 2, that’s exactly what the blog post was advocating for: Redundancy. They weren’t saying give up on Cloudflare (as they mentioned, they use it themselves).

They do blame you though, most people won't be aware of the real issue, when cloudflare went down, the trending things on twitter were #spotifydown and #applemusic

I suspect customers complaining on Twitter are not the ones cared about to whoever decides to use Cloudflare.

I build an app. I use CF for my app. Customers use my app. CF has an outage. Customers don't care why I'm down, they care that my app isn't working.

What is your solution for never going down?

> However, I believe it is somewhat childish and uncalled for to bash another company, because of a mistake.

The thing that they're bashing is not the mistake, which happens to everyone. They're bashing SPOF:

> EasyDNS was unaffected because while we do use Cloudflare to soak up large DDoS attacks against our nameservers, we don’t use them across all of our nameservers. I think somewhere in my book I wrote “DNS providers have a near-pathological aversion to SPOFs” (Single Point of Failures). Maybe only we do.

Well if you are going to rant, you might want to rant and quote what they ACTUALLY said, not what you conceived in you mind what they were saying

You indicate they said " "I use easyDNS so I didn't notice it at all tbh" but NO WHERE IN THAT ARTICLE was that statement

The actual quote is

"We’re familiar with Cloudflare’s DDoS service for DNS providers, because we use it ourselves. Fortunately easyDNS was not impacted by the outage (I didn’t even notice it, tbh),"

This is a MUCH different statement than you attempt to cast out as call "childish". He is stating that the services they use of CloudFlare was not impacted by the outage

I guess this is off-topic, but the stock photo they're using is cracking me up.

Guy at work... coffee cup on a tablet he's using for a coaster... except he's also drinking whiskey from a beautiful crystal glass... there's a folded paper airplane... there's just so much to unpack here, it's pretty hilarious.

> except he's also drinking whiskey from a beautiful crystal glass

And he has left the stopper off of the decanter like some sort of animal.

I think we are supposed to believe that the person here was just dicking around online, fiddling with paper, finishing his morning coffee, when all of a sudden he gets an email asking if anyone knows what's going on with the website.

So he stops what he was (not) doing, puts down his coffee, and starts poking around. At which point he realizes he needs something stronger than coffee. As he is pouring his glass he is confronted with the true horror of the situation, drops the stopper on the floor and just holds his face wondering why the Universe hates him.

I wonder if we can get JJ Abrams to option the movie rights.

> We call it Proactive Nameservers, and we’re the only company in the world doing it for some reason.

Wait, why? [0]:

> Proactive Nameservers is a patent-pending system that optimizes the nameserver delegation for your mission critical domain names.

Oh.

[0] https://easydns.com/dns/proactive-nameservers/

The solution is the Decentralized Web (DWeb), such as IPFS (https://ipfs.io), Freenet (https://freenetproject.org), GNUnet (https://gnunet.org) or Hypercore (https://hypercore-protocol.org). We should start using them to avoid centralization and embrace freedom.

Can you explain a bit more how this is a solution? Cloudflare isn't facebook. They have plenty of competitors, they don't have a massive moat, and they are almost exclusively used by businesses. Despite all of this, we should ask ourselves how we even got here. Why would companies move to DWeb, when they are already choosing to use Cloudflare instead of Fastly/Cloudfront/Akamai.

No salespeople to sell dweb

How do I publish a website on one of these?

https://www.dgendill.com/posts/technology/2019-10-15-publish...

Can any of those handle massive scale?

If more companies are willing to provide the same level of service and price as Cloudflare, then they can get in on the game, too.

This is just an advertisement for easydns.

Bingo. Still a good read, and easydns is just trying to profit over a screw-up from a competitor, but still essentially an ad.

> but still essentially an ad.

Compared to the endless content marketing Cloudflare posts [1]? It's an ad, but they're still right. That's just good content marketing (informative, relevant, and perhaps you buy something because of it).

[1] https://news.ycombinator.com/from?site=cloudflare.com

The more people that use anything other than Cloudflare, the better. I had this conversation with people back in ~2010 about Facebook and they all ignored it. They will again, but this time the consequences of centralization will be even worse.

It won't just be one single website that goes shitty with blockages and manipulation and censorship. It won't even be just the web. When Cloudflare achieves their goal of deep packet inspection at every peering and transit point it'll be the end of the internet as we knew it and the slow transition to just another cut apart "China-net (tm)".

Whether this is an ad piece by a competitor or not, the problem with monopolies is that "the market" (if there is one) gets skewed incentives. Cloudflare has received heavy investment by FAANG (+MS) [1] before their IPO, so rather than eg Google or others with a vested interest and capability stepping up the game and invest into new IP control plane-level DDOS protection standards or similar, the situation smells more like a backdoor deal, such as an agreement to not go after a particular market segment.

Let's also not forget Cloudflare in particular have been accused to host/hide the very bad boys that make protection from DDOS necessary in the first place. Whether or not that is the case, a quasi-monopoly leaves customers with no choice.

[1]: https://petri.com/microsoft-google-and-others-invest-in-clou...

If only DDOS attacks were taken seriously and their perpetrators punished accordingly (and maybe if the network had better ways of self-defense) instead of companies and websites having to fend for themselves (or having to resort to solutions like Cloudflare).

I am not sure I understand this comment, in the context- cloudflare misconfigured some routes and it was quickly resolved. was this a DDoS?

It was not.

His point is why the internet has a single point of failure -> Cloudflare. They offer DDoS protection for free.

I think the point was that so many companies wouldn't have to rely on anti-DDoS protection from Cloudflare.

What I find really shocking is the abundant use of CF on piracy websites of all things. Not the serious ones of course, SciHub and library genesis are mirrored differently.

But a lot of small torrent websites and such simply won't load without JS and specifically CF code. It's pretty crazy. Luckily I don't use any of those bEcAuSe IlLeGaL but still, I find it really depressing, especially when webtorrent, IPFS etc are available, and frankly many of those pages will never have to bear a load that makes CF a requirement.

its not about caching or handling normal traffic. its about ddos protection. sites like that are frequent targets.

and there should be enough of them that that shouldn't matter.

Now that Cloudflare is also a registrar, they could pretty easily implement a nameserver failover like EasyDNS. I hope this event underscores the importance of that to them.

It's worth noting Cloudflare also supports secondary DNS, but only for enterprise customers: https://blog.cloudflare.com/secondary-dns-a-faster-more-resi...

Does anyone know a dashboard/list for past Akamai outages. Surprisingly, I haven't seen a lot of news about Akamai downtimes. I searched on Google and the last one I found in a news report is from 2011 when its customers like Facebook, Twitter were impacted.

They were a PITA to work with when I used them in the past but if they are really that good in service availability, you can have some justification for their overpriced service.

AFAIK Akamai only makes their service notifications available directly to subscribers.

Cloudflare doesn't work without JS, even for static jsless sites. It's a pest.

Well, I usually block it anyway, because it is not only a single point of failure, but also a single point of concentration of data, that can be used to track, spy on and profile users. As such, I do not blindly trust Cloudflare. I remain sceptical, no matter how positive their public image is. Especially I would not set my DNS to cloudflare.

Who do you use for DNS then that you do trust?

For lookups, it's not that hard to do your own recursive resolver.

I love all the comments about fail-over for DDOS/DNS protection. What is your budget? Well we are looking at around $0.00 for our maximum allowance. Ok so single point of failure it is I guess we are done here. Companies only say they care when there is a problem, the reality is that they dont.

It obviously takes more than $0.00 but not that much more. It’s a matter of adding a second DNS provider and making sure you replicate DNS records manually or with AFXR.

What’s really puzzling is if the same companies spend money on active/passive failover for application and database servers while overlooking DNS single point of failure.

Precisely. I remember when the CEO of my old company came to me and said (with respect to moving from a on-prem model to SaaS), "What's our SLA going to be?"

"Well, what do you want it to be?"

Give me a number and I'll tell you how much it will cost and how long it will take to get there.

AWS in 2012, DynDNS after that, now Cloudflare...I wrote about this a few years ago, the threat of the singularity of the Internet. What was distributed will be centralized again. http://tuxlabs.com/?p=430

Cloudflare is hell when you need to load scripts and css 403'd by them.

Here's my work around:

1) open developer tools 2) refresh page 3) move tab into a new window 4) find a blocked resource in the network tab of developer tools and open it in a new tab 5) verify humanity 6) repeat step 4 and sometimes 5 for each resource the page requires 7) move the problematic site out of the new window and close all of these tabs at once

It is a terrible experience. CF devs and publishers have no idea how inconvenient their service is. I wonder if they have ever lived in a region where every ISP is both incompetent and listed in the CBL?

I frequently skip sites that use CF. The captchas are obnoxious.

The entire concept of a webapp firewall seems a bit backwards to me. Developers should fix their insecure applications.

Turns out no one got fired for choosing Cloudflare.

When I was figuring out how DNS works back 10 years ago, I was told "Why don't you just use 8.8.8.8 everywhere? Why do you need your own DNS server for anyway?" every single time when I asked a very specific configuration question. Some years later 8.8.8.8 was just replaced with 1.1.1.1 with same critical counter questions instead of helping. So, it appears to me, instead of understanding DNS and routing, people and tech-bro businesses take shortcuts by using centralized infrastructure for their systems, creating dangerous configurations. Now, Cloudflare have made a mistake, and half the Internet crumbled down. See the irony?

"Turns out over half the Internet has a single point of failure called BGP."

Is it realistic for a small-medium sized business to have more than one DNS provider?

> Is it realistic for a small-medium sized business to have more than one DNS provider?

Yes: as the weblog post points out, you can have EasyDNS as your master with their multiple DNS servers, and then also have (e.g.) Route53 slaved to EasyDNS and have those in addition to EasyDNS in your records.

DNS servers have had replication for decades.

It's easy enough if you sign up with a DNS provider provider.

Question: is it even possible to have DDoS protection without using a provider of it which becomes a single point of failure? Or is it maybe possible to decouple this single feature from everything else that Cloudflare provides that could take out all the sites in the future from an unrelated misconfiguration?

I don't see the centralization as a positive, but I'm wondering what percentage of the websites that were taken offline see themselves as having no choice but to use Cloudflare in order to prevent themselves from being taken down anyway from malicious actors instead of by accident.

I think you could use Cloudflare as your primary DNS provider and benefit from their DDoS protection, but also specify backup DNS name servers with a different DNS provider in case Cloudflare fails.

Sure, Arbour and others sell devices to deal with ddos and many isps have clusters of these which you can use. Of course this is a service that costs money.

Starting yesterday, I have been getting Cloudflare's hCaptcha image identification quizzes on every site that uses them. Googling the issue indicates that the IP address range may have been blacklisted, if say my laptop (a Mac that doesn't visit any shady sites) or my router has been compromised and is now running a botnet or something.

It seems unlikely because my phone can access sites on wifi fine, and again my Mac doesn't appear to have any malware.

Could it be that the outage is somehow relate to Cloudflare putting these captchas up as a precautionary measure?

It's interesting to me that Cloudflare doesn't really have any competition with a similar business model. I suppose the free plan requires quite a lot of spending before the upgrades offset it.

I don't need a free tier. I just need a "basic bitch" tier for my personal usage. Who are some Cloudflare alternatives for this?

Depends on what you want but a very solid free CDN and DNS option is: host your site on netlify and use dns.he.net for your nameservers.

Another good DNS option is dnsimple.com or, indeed, EasyDNS. For even more redundancy, use one provider as your domain registrar and another for your nameservers (and set short TTLs for your zones so you can re-point IPs quickly if you need to).

For the other things Cloudflare offers on their free tier, I'm not sure what good alternatives exist (there must be some, I'm just not familiar with them outside of the obvious AWS alternatives).

Edit: one caveat with above advice, I have no idea if netlify use cloudflare behind the scenes...

Edit 2: For other options, checkout https://www.cdnperf.com/ and https://www.dnsperf.com/

If all you're doing is hosting and CDNing static content, the US cloud providers can do this very, very cheaply.

There are other large CDNs like Akamai. They just don't compete with Cloudflare in the consumer sector. It probably doesn't matter for them because enterprise contracts are where all the money is at.

Sucuri seem to have pretty similar business model, which is a proxy in front of your site that handle security and cdn.

I meant the "CDN with a meaningful free tier" part of the business model. That's why "half the internet" has CF as a single point of failure.

AWS, Azure and Google Cloud all offer competitive alternatives for the small business sector. Personally, I find the actual cost of CloudFlare very difficult to reason about in advance of actual use. This is in part because it's easy to miss a bit of the a la carte model you need.

Full disclosure, I work at Google on the SRE team for Cloud CDN. If you want a credible alternative for the CDN part of Cloudflare, our product is extremely fast. We were all very sad for Cloudflare and watched the whole affair closely, as we've got a few customers that use our services alongside Cloudflare's.

The jist of my comment is why nobody else has tried the "liberal free tier" model for a CDN, as it seems to be working for CF.

Sure, but all 3 cloud providers have free tiers for this stuff, don't they?

Not that would work for a CDN use case as far as I know. The cloud providers are notorious for high egress costs.

The pricing for Google Cloud CDN doesn't look outrageous to me.

> But if you want to use a preferred DNS provider, such as Cloudflare, who use their DNS responses to optimize your website proxy. That works best most of the time, so then you want to go with an active/passive model that will step back when things are going according to plan, and then when these periodic network cataclysms do occur (and they will), they step into the breach and update your nameservers so that you at least stay up until the crisis is over.

Copy editors are cheap and your reputation shouldn't be.

Who at a C-level position is going to tell anybody that the potential risk of Cloudflare (or Amazon/Azure/GCP) going down should be protected against?

I would applaud them, but I wonder.

There’s a lot of truth to this. Cloudflare for now has achieved IBM status in that no one will be fired for choosing Cloudflare, in spite of any issues that arise.

My guess is Wall Street HFT or other financial areas with very strict unscheduled downtime penalties where they are effectively incentivised to be batshit paranoid as it would take decades for any penny pinching to remotely pay off. I don't know if many of them use Cloudflare for their domains though.

This is the 3rd DNS related outage I've seen in a week. Frountier Communications lost their DNS. Trying to recall what the 3rd one was.

My Unbound resolver round-robins DNS-over-TLS requests, between Cloudflare & Quad9. Cloudflare's outage never impacted us that I could tell.

Nevertheless, I am reminded that I ought to add a couple of DoT providers (who aren't Google). Not sure who else came online since I setup.

High availability is an insurance game, and perhaps we need to start treating it that way.

Rather than admitting that your customers need to maintain a business relationship with your competitors, you need to admit you need to maintain a business relationship with your competitors. That we need a moral equivalent of underwriting in the cloud space.

I am not sure, if I would trust anyone who misuses the term "Single-Point-of-Failure" on matters of reliability.

Why not? It's a single thing, that if it fails, causes your app/website/whatever to fail.

Because it isn't a single thing, it is a redundant system. Redundant systems can also fail, that doesn't make it a _single_ point of failure.

You can add another system in parallel as the vendor of the product suggests, or you can improve the resilience in the redundant system.

To make a hyperbole: my galera cluster is failing, its a single point of failure, so I setup a cockroach cluster in parallel.

In a way, it is right, as there are failure modes specific to the individual systems, but I think, it is incorrect, to label that a SpoF.

That is a fair point; SPoF depends on what level you're looking at. A RAID array removes the SPoF that is a single disk, but still leaves a SPoF in the RAID controller or CPU or power supply; a ceph cluster can withstand the loss of a whole rack but could still fall to certain software bugs. Likewise, "cloud" companies are internally redundant, right up to the point where they aren't. It depends on how you scope the question.

Instead of trusting any DNS provider with your queries, I recommend running your own recursive resolver with Unbound.

https://nlnetlabs.nl/projects/unbound/about/

But then aren't you just trusting your VPS provider or ISP instead?

Unfortunately, ISPs can see the domains you connect to, even if you use DNS-over-HTTPS.

https://irtf.org/anrw/2019/slides-anrw19-final44.pdf

Maybe it would make sense to have multiple independent sections of backbone at the BGP level. Instead of having one public AS/backbone, break it down into regions at least so that it is more confederated.

Similar problems are happening with crypto - yeah it’s distributed but so many people are using Coinbase that if they go down it’s going to cause a lot of problems

If you keep much of a website simple - plain html and css - won't that reduce the need for a CDN in the first place?

Yes, but cloudflare reach is much deeper then that.

I am not a cloudflare customer but my all websites failed that day. The reason was digitalocean uses cloudflare, I use digitalocean. So apparently I depend on cloudflare.

> Turns out half the internet has a Single-Point-of-Failure called “Cloudflare”

The other one is called AWS.

So in the end, who cares about single point of failures?

Militaries. The internet was created so the US President could command the US military even in the case of nuclear war knocking out many of the internet's nodes.

There are some (research) work on proof-of-human protocol, interesting direction

When Cloudflare causes the problem they were built to solve.

The other half is on AWS?

you can warn people, but they don't listen...

What responsibility and reparative measures has Cloudflare taken for Friday's incident? Was anyone fired for the mistake?

Why would firing someone make you feel like reparative measures have been taken?

Reparations would show an actual sense of responsibility. Firing someone would be appropriate if they were negligent. Other measures might be more appropriate. Is it enough that any time there's a Cloudflare incident, all we get are lengthy blog posts and sorries from Cloudflare?

Firing people for making mistakes is just going to foster a culture of secrecy and shame. Being so quick to fire is not how you retain talent and it isn't how you foster a healthy, blameless development culture in your workplace.

Firing people for making mistakes is a great way to watch productivity dive. The easiest way to prevent mistakes is to do nothing at all.

I understand the point being made here, but what are those affected supposed to take away? Cloudflare made a mistake that caused (x millions of dollars of lost online commerce revenues, y number of missed telehealth sessions, etc.) and since we do not punish mistakes, nothing was done. Sorry everyone!

Understood. Typically, as a company, you write - 1) what went wrong 2) how did it go wrong and 3) what you have or will put in place to prevent it from ever happening again

It's a learning process for all involved, really.

From the affected parties point of view, well, they should diversify their network a bit better. End users should hold those companies feet to the fire, not Cloudflare's.

> Firing someone would be appropriate if they were negligent

Maybe, but making mistakes is far from negligence. Besides which, if a single person can accidentally break your system, at least at Cloudflare's scale, that's an organizational failure, not a personal failure.

Pretty poor form calling out a competitor for something like this. Not the first time a DNS provider has done this, and won't be the last.

Reply on news.ycombinator.com