Hacker News Reader

Comment by tracyhenry | Hacker News Reader

And HackerNews seems to be much faster as soon as FB is back up. Not sure if this is just me.

No, you're not the only one, I noticed the same thing. In fact, immediately after reading and checking that FB is up I came back to the HN tab, reload it and it was way faster compared to a few minutes ago when FB was still down.

Telegram was down for me too

WhatsApp is still down

Something else is actually going on…

That doesn't mean something else is going on. It means that there are a lot of us who are web addicts who were temporarily deprived of our Zucker-smack and desperately flailed about in the rest of the web for a fix until our main dose got restored.

As for WhatsApp, that is of course a FB property.

I'm still thinking it's BGP poisoning of some top platforms... no one really knows what's going on, but BGP isn't byzantine fault tolerant

https://arstechnica.com/information-technology/2021/10/faceb...

Same.... was it a global DNS issue or was HN overloaded from everybody looking for updates?

Surely the moment facebook comes back up would be higher traffic as people rush to comment on it (see this thread's rapid rise).

Best guess: there's a DNS dependency in HN's critical path somewhere and the DNS server they used was being hammered with the facebook dns lookup retries.

Nah, unauthenticated HN was working very fast. It was simply the amount of traffic HN got while FB was down.

Here's a crazy idea: maybe a lot of HN people also spend time on FB during the day. When FB was down, they had more time to spend on HN!

Lots of facebook employees with nothing to do. We have this impression of HN being a huge internet property because of the importance of the news here in our lives, but in the great scheme of things, probably it has pretty modest traffic. 60.000 more people hammering it probably is a good load.

From a cloudflare blog post on this, their public DNS resolvers were getting 30x their usual traffic. Presumably that was the case for DNS resolvers everywhere, so that would explain a slowdown:

> But that's not all. Now human behavior and application logic kicks in and causes another exponential effect. A tsunami of additional DNS traffic follows.

> This happened in part because apps won't accept an error for an answer and start retrying, sometimes aggressively, and in part because end-users also won't take an error for an answer and start reloading the pages, or killing and relaunching their apps, sometimes also aggressively.

... > So now, because Facebook and their sites are so big, we have DNS resolvers worldwide handling 30x more queries than usual and potentially causing latency and timeout issues to other platforms.

https://blog.cloudflare.com/october-2021-facebook-outage/

Edit: Although looking at the graph on that page, it appears that only DNS requests for FB properties increased 30x, so that last sentence appears to be misleading. Regardless, a significant overall increase.

> their public DNS resolvers were getting 30x their usual traffic

But why does my browser need to do a DNS lookup every time I request a page from HN? It's not like the IP address changes every 5 minutes.

Short answer is because they've set their DNS TTL to 5 minutes: https://mxtoolbox.com/SuperTool.aspx?action=a%3anews.ycombin...

As for why, you'd have to ask them. It could be to allow for relatively quick DNS failover in the case of an outage. Or if using a proxy like Cloudflare, that provider could prefer short DNS TTLs to have flexibility in their routing.

Because hackernews sets their TTL to 5 minutes:

dig +noauthority +noquestion +nostats news.ycombinator.com @ns-225.awsdns-28.com.

news.ycombinator.com. 300 IN A 209.216.230.240

Cloudflare and similar companies only work if the TTL is very short. If they set it to some infinite number, they wouldn't be do even the most basic load balancing at the global scale in which they operate.

Once it came back people stopped mashing refresh on the related HN threads.

Perhaps more people visiting HN during the Facebook blackout?

The packets were piling up at the fork in the river

Yes. This seems to me to be true.

True for me as well. What would explain this? Does hn use some tracking from FB? Not trying to throw shade, but this is a surprising correlation.

Could it just be people pounding HN for any update on Facebook?

That's what I saw claimed, but that wouldn't go away the very second Facebook came online again

A non-trivial percentage of HN visitors also read FB. Absent FB, they likely spent more time on HN.

All the devices on the planet with Facebook owned apps installed having no concept of backoff retries and people constantly refreshing trying to figure out why Facebook isn't loading was pegging the crap out of DNS resolvers, slowing down queries for other legitimate requests too.

Once they came back online and their DNS was resolving, it alleviated all that pressure.

Maybe HN servers share room with something that got pinned down with requests when FB was down? E.g. 100000s of devices doing a login retry loop for some app since FB auth. doesn't work, or something.

DNS resolvers worldwide being bombarded with retries from clients trying to reach FB.

I did a little testing while Facebook was down. I copied an HN frontpage request from the network traffic tab in chrome’s developer console. Using the ‘copy as curl’ option, you get a fully formatted curl request in your clipboard that makes all of the headers and protocols of the original browser request.

I then executed that request in the terminal. This request was met with the same ~5 second delay I was seeing with my browser. Front and center was my authentication cookie in the header, so I removed that from the command and the request was normal in response time (extremely fast in HN’s case). I put the cookie back, it was slow, I took the cookie out it was fast. I did this 4-5 times until i was satisfied that was it.

The delay could have been due to upstream dependencies on DNS, extra load on the authenticated request pipelines, some Facebook subterfuge in the bowels of HN or just an extremely unusual syncopation in the patterns of the universe.

But clearly the authenticated requests were slow and the unauthenticated ones were not.

Probably Facebook trackers. I noticed a lot of apps (including my own) having major slowness on initial load while they were down. My assumption is it was their tracking code timing out.

But HN doesn't have any facebook trackers

I wonder how sure we can be of that.

Hit F12 on your browser and look at the network traffic. HN only loads a handful of files: the html page, a style sheet, a JavaScript file and a few GIFs for the arrows and faveicon.

The JS is uncompressed and very readable. It only contains logic for site use, no code to call back some other trackers.

A perfect example of site that doesn't need to use client-side analytics to sell ads.

Cool, cool. Try using your imagination though.

Well, these days server side api… er Conversions API is the way to go. This way if the user blocks the browser pixel the data can still be acquired for certain events, say a purchase. Google Tag Manager server side has something similar.

It was probably due to the global DNS system being overloaded due to people and apps repeatedly retrying facebook connections. Cloudflare reported 30x normal DNS requests for FB properties.

https://blog.cloudflare.com/october-2021-facebook-outage/

Yes, from New Zealand it is now normal speed.

I'm experiencing this as well.

Reply on news.ycombinator.com