Hacker News Reader

Comment by curiousgal | Hacker News Reader

I don't understand that logic, wouldn't people interacting with the website normally also generate the same amount if not more DNS requests?

View on HN

It will have been cached at closer to the edge, but once the TTL expires, so does the cache. That means all the DNS requests that would have been served via local caches end up hitting the upstream DNS servers. For a site like Facebook that will be creating an asbolute deluge of requests. Andecdotal but the whole of the internet feels sluggish atm.

Anecdotally, my personal website feels faster than normally. Gandi DNS.

No, since the positive response will normally be cached for "some time" dependant on a number of factors. The negative response on the other hand often won't get cached, again, dependent on settings.

Yep, I always make it a point to cache cache-misses in my code.

So then when I'm on some kind of blocked WiFi and nothing resolves, and I switch to a properly working WiFi your code will continue to fail?

It's not so simple to cache misses - you don't know if it's a real miss or some kind of error.

For example if Facebook cached the miss, then even when they are back up nothing would connect.

It's remarkable the effect even short TTL caching can have given enough traffic. I recall once caching a value that was being accessed on every page load with a TTL of 1s resulting in a >99% reduction in query volume, and that's nowhere near Facebook/internet backbone scale.

yep, prepriming the cache rather than passively allowing it be rebuilt by request/queries can also result in some nice improvements and depending on replication delay across database servers avoid some unexpected query results reaching the end user.

In the past I was the architect of a top 2000 alexa ranked social networking site, data synchronization delays were insane under certain load patterns high single low double digit second write propagation delays.

Yes. I handle around a million requests per minute. I exponentially increase the cache period after subsequent misses to avoid an outage ddos the whole system.

This tends to be beneficial regardless of the root cause.

edit this is especially useful for handling search/query misses as a query with no results is going to scan any relevant indexes etc. until it is clear no match exists meaning a no results query may take up more cycles than a hit.

I'm talking back-end not in app data caching. I would also cache misses there as well but with less aggressive ttl.

Negative responses are cachable with the appropriate time to live from the Start of Authority record for the zone.

I know you're just replying to the parent statement but unfortunately in this case the SOA went down with the ship. None of the (admittedly few) clients I've tested are caching the lack of a response for facebook.com's SOA or address records.

It's disappointingly common for cloud-backed apps and device firmware to go into a hot retry loop on any kind of network failure. A lot of engineers just haven't heard of exponential backoff, to say nothing of being able to implement and test it properly for a scenario that almost never happens.

Even if you assume Facebook's own apps have reasonable failure logic, there's all kinds of third-party apps and devices integrating with their API that probably get it wrong. Surprise botnet!

Normally the request resolves then gets cached locally, on the edge, by the ISP, … DNS is cached to a ridiculous levels.

But if the request does not resolve there’s no caching, the next request goes through the entire thing and hits the server again.

There's a lot of caching involved in the chain of requests that would alleviate this request volume if things were working.

My best guess is that after n many attempts to access the provided IP, the local DNS cache deletes the entry causing a miss. Then the cycle continues.

Reply on news.ycombinator.com