Hacker News Reader

Comment by belltaco | Hacker News Reader

https://status.discord.com/ is down. Wow.

And that is why you host your status page on separate infra.

We're dealing with a deeper level problem here. Since a lot of the internet is relying on Cloudflare DNS at some part or another, even many backup solutions fail. Since so much of DNS is centralised in so few services, such outages hit the core infrastructure of the internet.

A sudden disruption on a large number of services for everybody at once doesn't look like a DNS problem to me, with all the caching in DNS. It would fail progressively and inconsistently.

given that TTLs are usually very short now if your DNS server is configured correctly then caching shouldn't make a bit of difference.

I looked at the example above, all of patron, digitalocean, coinbase and gitlab (haven't checked the others) have a ttl of 1h.

DNS absolutely was an issue. I changed DNS manually from Cloudflare's 1.0.0.1 (which is what DHCP was giving me) to 8.8.8.8 (Google) and most things I'm trying to reach work. There may be other failures going on as well, but 1.0.0.1 was completely unreachable.

I don't use cloudflare DNS but google DNS and got the same problems thant everyone else

The problem seems to have been resolved now, you might have made the change when they fixed it.

> I don't use cloudflare DNS but google DNS and got the same problems thant everyone else

Cloudflare is also the authoritative DNS server for many services. If Cloudflare is down, then for those services Google's DNS has nowhere to get the authoritative answers from.

Except the services mentioned in the original post have a ttl of 1h. Unlikely they would all go down at the same time.

status.discord.com 5 minutes

No, I changed the setting back and forth while it was down to confirm that the issue was that I could not reach 1.0.0.1. All the entries I tried from my host file were responsive (which is how I ruled out an issue with my local equipment initially and confirmed that it wasn't a complete failure upstream -- I could still reach my servers). Changing 1.0.0.1 to 8.8.8.8 allowed me to reach websites like Google and HN, and changing back to default DNS settings (which reset to 1.0.0.1, confirmed in Wireshark) resulted in the immediate return of failures. 1.0.0.1 was not responsive to anything else I tried.

Again, it may not have been the only issue -- and there are a number of possible reasons why 1.0.0.1 wasn't reachable -- but it certainly was an issue.

pr0 tip: dont set your DNS TTL to 5 min (status.discord.com does)

Yeah, but then your status page provider switches THEIR provider, and suddenly you are on the same infra again.

It's hosted by statuspage.io, and their own status page was also down (metastatuspage.com). It is now back up, but their page shows the outage.

Status turtles all the way down, it seems!

Works for me.

I get a DNS servfail when resolving that DNS record, and many others:

    Server:    8.8.8.8
    Address:   8.8.8.8#53
    
    ** server can't find status.discord.com: SERVFAIL

So it's either not just cloudflare, or all those sites use cloudflare to host their DNS.

The latter.

    $ dig +short discord.com ns
    sima.ns.cloudflare.com.
    gabe.ns.cloudflare.com.

8.8.8.8 is Google and is down in addition to Cloudflare. 8.8.4.4 was still up.

8.8.8.8 is a caching resolver and it wasn't down. Do you understand how caching resolvers work?

Same. It has a big banner saying CloudFlare is down.

Works for me too... Australia

Reply on news.ycombinator.com