DNSSEC Outage on www.cloudflare.com – 2019-03-21 | Hacker News Reader

DNSSEC Outage on www.cloudflare.com – 2019-03-21

ianix.com

73 pointswglb35 comments

This blip was caused by configuration changes we made to our marketing site (www), which is distinct from any control plane (api/dash) or edge services provided to our customers.

Specifically, we were breaking out our "www" subdomain as a separately managed zone, and some of the configuration wasn't properly migrated.

This resolution failure was regrettable, but there was no impact to the service we provide for our customers.

Somewhere tptacek is in tears, and laughing.

https://twitter.com/tqbf/status/1116112320567087104

Just like you can f* up HTTPS when you have Strict Transport Security, you can f* up DNSSEC. While I don't really like current DNSSEC implementations, this is not DNSSEC's fault really. The idea I do like, that recursive resolvers can't tamper my resolving - that the recursive resolver is left out of the trust chain.

I can't wait to see what disingenuous statements he makes about a config failure.

DNS delegation errors are kind of rare. There are so many configuration errors in web deployment, and the most common errors I see which causes websites to be inaccessible are misconfigured certificates where:

• You get the default certificate for the web server rather than the correct one for the vhost.

• You have a redirect from www->bare or bare->www and the redirect vhost don't have a certificate.

• The certificate don't include the www subdomain name.

• A wild card certificate doesn't include the bare name.

• There are multiple web servers listed in DNS but only a subset has a correct HTTPS configuration.

We really need to push back against centralization on a small number of providers. As we have seen with previous Cloudflare issues and the big Dyn incident, a single issue can basically make the internet unusable.

I'm also really worried about that. (It's just a feeling but..) for a service as big as Cloudflare I think it may be not as failure proof and clustered designed as it should be for a massive DNS provider.

I for myself pay for a smaller DNS provider like CloudDNS with a bigger redundancy in NS servers (4 instead of 2 at Cloudflare) than going for free at Cloudflare.

> bigger redundancy in NS servers (4 instead of 2 at Cloudflare)

I don't think this is really all that important, as typically all these servers are running anycast. If you really wanted redundancy, you would have NS records from at least two different providers.

That's a good point. A backup DNS on another service is on my bucket list once sites get a little bigger and more relevant (right now it's not worth it).

You're paying for a service that's less robust because they have more IP addresses serving their database?

For the time beeing I'm not sure which is more or less robust because I have not seen any DNS outages in both.

Cloudflare is one control plane, but it's 1) anycast 2) over 100 locations 3) multiple servers per site. The SPOF with Cloudflare is either your account getting hacked, or (much less likely) a CF control plane/configuration outage.

Someone could probably build a better DNS provider than Cloudflare, at least for their specific site, but it would be difficult. I don't know of any other DNS provider I'd consider more robust, particularly against high load, DoS, etc.

(Disclaimer: worked for Cloudflare 2014-2016)

This is rather weird claim. Getting your account hacked is the least of your worries, when every single thing is maintained by a single organization, a single team, on a single autonomous system, with lots of reliance on single hardware and software components. This is basically three nines level stuff, with hours of downtime per year on average. It's relatively easy to do better than that with do-it-yourself CDN using DNS-routed multiple independent hosting providers for edge nodes, getting to four, five nines level. (and DNS separately is not worth talking about, because it has built in redundancy, anything will always be more reliable than a centralized DNS provider)

> with lots of reliance on single hardware and software components

Are your saying you have inside knowledge and that's the architecture? (I'm guessing no)

I've never seen any company use completely different software or hardware for each redundant instance of a service, so it's a pretty good bet CloudFlare isn't doing this either.

That would imply a SPOF. One component of any kind that can take down the whole thing. The only such “component” I can think of is a human, not a technical one.

I meant single kind of component, not one thing handling everything.

If you use juniper routers, for example, a juniper specific software, hardware problem or a mistaken assumption about juniper specifics could take down all routers and an entire network.

Usually any kind of such issue wouldn’t take out every component of a type unless user error is involved. Like deploying the wrong policy to all devices. Which again comes back to humans being the weakest link. I can’t really imagine a particular hardware problem that would kill all components at once. This being said I probably didn’t experience all possible problems so far :).

Sure, humans are the weakest link, who cares? Regardless of the cause of the failure, catastrophic cascades are more likely if the systems and operations involved are homogeneous. That's why evolution prefers diversity.

Natural evolution. Because it follows vastly different rules. I still fail to see any example given for how one “cascade failure” takes down all the routers in an infrastructure like Cloudflare’s. I can think of a dozen ways a non-homogeneous system can fail (like incompatible updates and config, confusions between the types of components, etc.). In the end you usually end up with massive expenditure that introduces just as many problems as it solves.

You may be onto something that nobody else spotted but so far the arguments are too generic (“failure”) or incompatible with the premise (rules of natural evolution applied to very artificial systems).

Don’t get me wrong, I’m happy to learn something new and understand a different point of view but so far the arguments don’t help me.

I actually used that example because they had this exact problem in the past, here's the link: https://blog.cloudflare.com/todays-outage-post-mortem-82515/

And Google had a similar issue recently where they didn't test their firmware upgrade auto rollback, so a boned firmware upgrade bricked a region until they could manually recover it. If they had rolled out to all regions at once, and all the devices were the same, it would have been chaos.

Often the only thing preventing mass upgrades from becoming mass outages is some part of the network just happens to work differently.

Well... sure. Just as long as you keep in mind that a "not really tested [something]" will very easily get replicated on multiple vendor devices. It's not like they did it because they hate Vendor1 but will make sure to test all the corner cases with Vendor2. Some scenarios will by design ensure you can't have all of them fail because they are not susceptible to the same failure modes. But to compensate you introduce additional failure modes (especially human error) by using a heterogeneous environment. Not to mention the expense.

I was actually hoping for an interesting human independent failure. A cosmic radiation caused bit flip that would occur on a particular device. :) Human error can overcome any setup.

Without saying too much, cf has a fairly diverse network in terms of hardware. Partly because the sprawl makes it hard to keep every single colocation running the same hardware, when a single colo can consist of hundreds of independent nodes (metals).

I agree that we need to push back against centralization. But I'm not sure this incident is why.

Where ever you host something, the provider may make mistakes which causes you to be off the air for some time. It doesn't matter if a big provider makes a mistake and takes down lots of domains or if lots of smaller companies make their own mistakes and over time (together) also take down a lot of domains.

If, as a user of the internet, you only care if there are cat video somewhere, then yes, having lots of small providers makes a difference. If you just need to visit one website, it doesn't matter how many other sites are down.

The Dyn incident is a different story. Because there a (ddos) attack on one site, took down lots of other sites. In that case using smaller providers would have helped. But one of the features cloudflare is selling is ddos resistance. Which is something only very big players can offer.

Hetzner and OVH (and presumably other cheap hosters) offer DDOS protection for some time now. Don't know if it's any good though, as the nature of a DDOS protection service makes it difficult to standardize and discuss publicly.

What I don't quite understand is how that 2016 attack was SO effective. I thought DYN's thing was distributed AS THOUGH you were not putting all your eggs in one basket.

Sorry I am not sure what you mean. You can happily setup your own DNS servers as of today and host any domain. You need only 2 publicly routable IP addresses. What is centralised in this context that you need to push back on?

"why should a DS record for www.cloudflare.com matter?!" - that was my exact question when I looked at the output from the DNSSEC Debugger.

Now I'm curious as to why www has NS records...

(DS records are used to tie DNSKEYs to a child zone from the parent zone).

It has NS records because that's how the owner of cloudflare.com (within the Cloudflare Dashboard) delegates control of www.cloudflare.com to another team.

Ah. That makes a lot of sense, thanks.

The title gives the impression that this was an outage of Cloudflare's DNSSEC service, whereas in reality the Cloudflare website was down due to a DNSSEC error. Ideally the title should be clarified to make it obvious that Cloudflare customers were not affected.

time doesn't match exactly but that might be it: https://www.cloudflarestatus.com/incidents/3k7gtg4fml1p (https://www.cloudflarestatus.com/history for the list)

The Cloudflare Android 1.1.1.1 VPN app was down most of the day.

Reply on news.ycombinator.com