Every device with FB app is now DDoSing recursive DNS resolvers
twitter.com
twitter.com
Hope they will just go back to fix FB and this is just in my head :)
I am also thinking that all the poorly coded sites that don’t work unless the Share on Facebook button loads are also going to hemorrhage money. So are all the e-commerce sites that rely on Login with FB.
I hope this results in everyone rethinking adding all that shit to their infrastructure before the next time.
OpenDNS: https://cachecheck.opendns.com/
Why on earth would any system need a TTL beyond 1 hour? The longer the TTL the longer your outage period will be if a cache is poisoned.
Anyway, the since the NS records for a domain come from the next level up, i.e. in this case the .com nameserver which are also the root nameservers, if they had a short TTL then those root nameservers would get hit pretty hard. Also, it isn't your choice what the TTL on your NS records is, it's admins of your top-level domains.
So, two days is the TTL you get for your NS records. Of course, it takes way more than two days for people to stop querying your old records. I gave up after 30-45 days.
Load. Lower TTLs mean shorter term caches and more traffic.
This is a big deal, especially for root servers -- which is why the TTL is not customizable.
> The problem isn't (wasn't) facebook.com's A records, it's that the
> authoritative nameservers for facebook.com are (were) unreachable.
[0]: https://serverfault.com/questions/479367/how-long-a-dns-time...
I thought so too, but at least some do cache according to Cloudflare's blog post:
> Consequently, 1.1.1.1, 8.8.8.8, and other major public DNS resolvers started issuing (and caching) SERVFAIL responses.
https://nlnetlabs.nl/documentation/unbound/unbound.conf/#inf...
https://matrix.org/blog/2019/03/12/breaking-the-100-bps-barr...
(I think this was posted already, and it's from 2019, but some people must have missed it - I only saw it this year)
https://en.wikipedia.org/wiki/Mobile_cell_sites
I had a friend who worked deploying them many years ago to events for one of the cellphone providers in NZ.
Well, you know what can happen / If you're not careful with fire. / And for the light on a cigarette / A rug feels rather too expensive. / And from the rug the fire, alas, / Might have spread to the whole house / And who knows what would have happened thereafter?
There would have been a fire in the district / And the fire fighters would have had to come. / They would have honked in the streets / And unloaded their pipes. / And they would have sprayed fire / And it would have been in vain / And the whole city would have burned with nothing to protect it.
And the people would have jumped around / Fearing for their possessions. / They would have thought somebody had started a fire. / They would have grabbed assault rifles. / Everyone would have shouted: "Whose fault is it?" / The whole country would have rioted. / And they would have shot at the ministers behind the lecterns.
The UN would have become involved / And the UN enemies as well. / To preserve peace in Switzerland / Both would have come with tanks. / It would have spread, little by little, / To Europe, to Africa. / There would have been a World War and humans would be no more.
I lit a match / And it made a fire. / And for my cigarette / I wanted to take the fire from the match. / But the match slipped from my hand and landed on the rug. / Thankfully, I picked it back up.
2a03:2880:f1ff:83:face:b00c:0:25de
You know (or could find out) the hashing algorithm. I’m sure you could come up with a model estimating “pollution” based on hash rate, kWh per hash, “greenness” of the energy source, and estimated hashes until a prefix collision.
Excited to see what you come up with.
I guess alternatively you could return garbage (127.0.0.1) with a 5 min ttl or so to get clients to backoff but also problematic.
Exponential backoff is a really bad antipattern that harms users. There are other, better ways to shed load.
Also, there's no evidence that any DNS infrastructure was overloaded during this event, so what are we even discussing?
And what about a five minute "blip"? Arguably it should resume proper operations as soon as the link goes back up, not some (unknown to user) time afterwards.
I use 0.0.0.0, though I'm not sure if some layer in that mess would interpret it creatively. Has worked on my machine for years at least:
$ ping facebook.com
connect: Invalid argument
Does DNS support extensibility?
If so, an option could be responding with a fake answer like 0.0.0.0 + some TTL, plus an extension that tells modern clients "hey, this is actually a SERVFAIL but please cache it for the TTL".
because of this there are lots of settings in most servers to ignore and sanitise shitty responses.
No! "that size" changes over time, so you have to be responsible all the time.
analogy: read up on the '-f' flag for ping
Logout which I believe they just forced for all users sends you to the cache and it'll be fast. dang@ mentioned it during the giant S3 outage a few years ago.
I've been doing ~SRE for 1.5 years and I've worked or helped on 3 outages related to negative DNS.. Please don't use negative cache, if you don't know how enough about DNS and can't monitor it
Your browser AND operating system AND router already provide DNS caching, it's not something average user should even think about. You might want to consider it when things in your ISP go wrong (hello BT), or majority of computer request the same domains frequently, but then again, your router should do it already.
"In summary, SERVFAIL is unlikely to be cached, but even if cached, it'll be at most a double- or even a single-digit number of seconds."
That would be fatal right now, wouldn't it? That would mean every major ISP's DNS server right now forwards millions of identical DNS resolve requests to the (currently null-routed) Facebook DNS servers. These must be millions, as heck, every larger website uses FB tracking tools, "like buttons" etc. Are they at least smart enough to throttle based on a domain/ip hash? Else it could happen that DNS servers of major ISP are soon overloaded as (constantly failing and thus uncached) requests to FB DNS would eat up all bandwidth/ressources?
Then you have port limits, usually each request goes out on a new port, a recursive resolver can only have 64k requests outstanding to any given authoritative (or upstream) server IP for each IP the recursive uses. Facebook runs with 4 hostnames listed, so that's a limit of 256k requests outstanding, 512k if your recursive does IPv4 and v6 (and 1 M if they're also making whatsapp requests).
DNS services for both domains appear to be back up by the way.
On the authoritative side, it's not too hard to manage this load. If you can't handle the big crush to start with, drop all requests, and then accept all the requests from 1.0.0.0/8, and add one /8 at a time as CPU permits until you're allowing everything. Once you handle the initial crush from a resolver, it should go back to normal load, and there should be some distribution of load across the various /8s. I wouldn't expect it to be evenly distributed, but it should be even enough.
Disclosure: I worked at WhatsApp, but left August 2019. I don't know anything about this outage other than idle speculation. I don't know if FB has a procedure to slow start DNS, but the theory is simple; the practice is complicated by the DNS ips being used in Anycast.
I think it is wishful thinking, because that would basically be caching which is not allowed by the RFC. In 2017 the BIND implementation changed to a default cache time of 1s which would certainly ease the problem.
> then you have port limits, usually each request goes out on a new port, a recursive resolver can only have 64k
I'm unsure if this helps or worsens the situation, depending if the 'collapsed forwarding'/1s caching is in place. If this is not the case, ephemeral port exhaustion would kick in, at which point the DNS server will not be able to server other requests.
> On the authoritative side, it's not too hard to manage this load
Of course not, all you need to do is just present any response which will be cached by downstream resolvers. No smartphone/end user device will query the authoritative side as long as there is just any (even stale) response.
You can use the same local ip/port to contact multiple server ip/ports, so filling up connections to FB ips shouldn't prevent you from connecting to others (but there are plenty of ways to do that wrong, I guess)
>> On the authoritative side, it's not too hard to manage this load
> Of course not, all you need to do is just present any response which will be cached by downstream resolvers.
You need to present a response before the resolver times out. One can certainly imagine a situation where the incoming packet processing results in enough delay that the responses arrive too late and are discarded. In the right conditions, this queuing delay would never clear and things just get worse. If it doesn't happen, great, but if it does, dropping most of the requests so you can timely handle the few you accept is a good way to get moving.
And to clarify a bit, the queries aren't "no longer being cached due to query failures", it's because their TTL expires and the resulting SERVFAIL from the next query (which fails) isn't cached at all.
btw; going trou the (dns-)logs of ppl you personally know is somewhat rude
why did the public user dns servers (isps/google/cloudflare/etc) drop the facebook records? dns is designed to be able to handle an unreachable server. (or was some mitigation for some sort of abuse added along the line somewhere that invalidates zones if they're associated with completely unroutable addresses?)
FB's eggs were all in one basket, and the basket broke.
But even if they did hardcode an IP, the underlying infrastructure for Facebook was also down not just DNS resolution of facebook.com. So even if the FB app didn't need to resolve a hostname, it would still be broken.
DNS is a translation from human readable addresses to machine addresses.
BGP determines how to find those addresses from your server to theirs.
your browser asks your computer which asks your router which asks your isp which asks .com's dns servers which ask facebook's dns servers for facebook's ip address.
each layer will cache the results so say, even if 100,000 people in seattle want to know facebook.com's ip address, only the 5 or isps who provide internet in seattle have to ask for facebook.com's ip address, so 100,000 requests, but only 5 actual requests.
even the per-device and per-home cache is helpful, because 500 page loads in 15 minutes still only results in 1 actual dns request.
Here's the issue:
Failures aren't cached.
so while 100,000 people in 1 second trying to get facebook's ip only resulted in 5 requests going to the core dns servers, now results in 100,000 requests in 1 second trying going to the core servers.
i thought so too but according to Cloudflare's blog post, at least some of them do:
> Consequently, 1.1.1.1, 8.8.8.8, and other major public DNS resolvers started issuing (and caching) SERVFAIL responses.
AND BROKE THINGS!
Recursive DNS is pretty easy to do for really large volumes on a $600 1U server. It's not like the days of 15 years ago...