Ask HN: Why is HN so often down?

I've been frequenting this site daily for over 15 years now, and I don't think I've had error messages more than a handful of times in that decade and a half. Today was indeed one such time - which is what triggered me to check out this particular post. But at least in my experience it's not what I'd call "often". Maybe you're just very unlucky, or perhaps I'm extremely lucky... Who can tell?

I've been keeping track, it goes down often (checking from US East, US West, Japan, Australia, and NL)

https://hackernews.onlineornot.com/

here's what it looks like on the monitoring side:

https://x.com/RozenMD/status/1763152885892727188?s=20

(HN is slightly more stable if you're logged out)

I believe it's just a virtual machine and IP addresses are not anycast; no need to check from multiple locations when you can reach the host and the issue is in it returning HTTP 5xx.

True, I mainly monitor from everywhere as a means of dogfooding my own service (and just incase HN gets replicas)

https://hn.algolia.com/?dateRange=all&page=0&prefix=true&que...

HN runs on a single core, I doubt they'll get replicas

Do you have a longer term graph please?

I could make that happen, yeah

For the past couple of months I've had these errors multiple times a day, it's definitely often down now.

Months? I'd say I'm mildly addicted to HN and I noticed this week it was hit or miss, but only this week. Very surprised to hear you say months.

I confirm that “months” seems the right time range for me (accessing from Brazil).

I open HN, idk, 10 or 20 times a day, I would guess I see the error message around 5 to 10 times a week.

Do you have a VPN you can try on? I run a VPN connected to a San Fran server 24/7 on my mac and I very very rarely have issued with HN (or any internet issues at all tbh).

I get it happening every few weeks for a short amount of time it seems. More so recently (also been a site user for approximately 10 years).

I had never seen it for more than a decade, but in the last couple of years it has become very frequent.

Maybe somebody bumped the desk: https://news.ycombinator.com/item?id=23472870

It is not often down, but it's very easy to hit its rate limiter when upvoting or commenting. Waiting a minute usually fixes it.

The rate limiter is even stricter when not logged in — for a period I wanted to take a break with getting into online arguments, so I stayed logged out, which would cause my IP to be banned after a day or two (a bit annoying, though easily unbanned with the self-service system: https://news.ycombinator.com/item?id=4761102) — IIRC dang confirmed that it would be better to stay logged in to avoid this anti-bot measure.

I guess that's the price to pay to have such a popular website that's maintained by a single person.

I really wish the rate limit was relaxed when favoriting posts specifically. Favoriting a post counts as two requests since it redirects you to another page, so you'll often hit the rate limit page while favoriting, but the favorite still goes through because only the redirected request gets rate limited.

"I guess that's the price to pay to have such a popular website that's maintained by a single person."

Why do you think, it is only one? There is definitely help with the moderation for example.

I can't find the source, but there is an article that mentions the entire site is moderated by 2 people. Maybe they've added a third since then.

I think the 2nd person in the article you read was Scott Bell, who stopped working on HN around 2019 [1]. Still, dang isn't the only moderator here [2], just the only one who posts publicly.

[1]: https://news.ycombinator.com/item?id=25055115

[2]: https://news.ycombinator.com/item?id=25235720

Ah thank you for the clarification!

I've had that same message a couple of times over the last few days, while logged in (EU).

It's kind of expected behavior when there's a thread with thousands of comments (like when OpenAI shafts their CEO [0]), but the last few days seemed relatively calm.

So maybe the server is crumbling under the increase in traffic from higher HN rankings in Google [1]?

[0]: https://news.ycombinator.com/item?id=38310213

[1]: https://news.ycombinator.com/item?id=39423949

It was down for me a bit today.

But it's a great free service with no tracking/ads so I'm fine with that.

I agree it's a great free service and I've rarely seen downtime, but as a nitpick:

The posts for YC backed companies that are hiring (where the comments are turned off) are ads.

I believe most regulars know who our sponsors are :)

If you put it that way, the whole site is a giant ad for Y Combinator. It just happens to be a public service too.

It's also one of the few sites* i frequent where uBlock Origin says it's not blocking anything.

* Tbh the only other is Ars Technica, and there only because I'm logged in as a paying customer.

I'm in Ghana, and can confirm that I'm seeing this a lot recently. Sometimes the problem persists even after multiple refreshes.

I'm almost always logged in, and it happens even when I have not visited HN in some hours.

I've noticed it too - also in Europe. It could be a result of increased traffic from a recent Google search algorithm update which increased HN's ranking significantly. I understand HN infrastructure isn't enormous, so it would make sense. HN ranking discussion here - https://news.ycombinator.com/item?id=39423949

Dang said they're working on it, I presume they are upgrading the hardware soon or something... (edit: in any case they are aware and it hopefully shouldn't be an issue long term)

He’s also been saying that for years, so.

Maybe they have improved performance but traffic increased even more?

There was this recently:

A steep rise of Hacker News in Google rankings https://news.ycombinator.com/item?id=39423949

Traffic has been increasing for years, so a multi-year "working on it" effort is probably a good thing - it shows an ongoing effort to size the resources to the level needed at the time, and adjusting as times change.

Is that a reason to believe it less? You can also see from hiring threads that it's a very small (but growing) team and that inherently it is truly being worked on.

Hence, I would conclude that he is working on it for years.

Traffic has probably been going up for years..

I was under the impression that a lot of the code is single-threaded and so throwing hardware at it won't do a whole lot.

Sheesh. Isn’t this the easiest type of application to just cache the reads and queue the writes, then you can scale horizontally easily.

Same, it seems HN is getting more traffic but hasn't been keeping up with the load.

Which is interesting because YC has funded many infra companies.

I usually browse through hn.algolia.com and that has a much higher uptime than news.ycombinator.com

In 2024, it is surprising how often HN is unavailable.

It's still just one old server. Which itself is impressive, but with the search indexing changes maybe not enough anymore?

I’m new but been checking HN for the past weeks daily, multiple times a day usually. I haven’t experienced this on any single of my visits. Weird.

I've had mixed results. Some visits were fine, especially those where I wasn't logged in. During others I've seen errors, often when writing comment replies or voting.

Just guessing but I believe they run their own colocated infraestruture and don't have much or anything that autoscales, so if they're not overprovisioned enough and there's a lot of threads that have too many comments it happens. I also saw they are hiring for an infra engineer role so they might also be understaffed.

It is a single one-core server, and I believe it's pretty old.

Consider the 'HN hug of death' we have patience for when a site linked on the front page is down.

HN must be getting orders of magnitude higher traffic than that constantly.

I remember I read somewhere here that it is recommended to log off your session when you do not need to stay online to comment or post something.

No, it's the opposite, see: https://news.ycombinator.com/item?id=39548152

EDIT: I guess confirmation from dang isn't proof enough. Alright.

A much more recent comment from dang asked people to logout if they could when HN was under stress as it would make it faster for the user: https://news.ycombinator.com/item?id=38310213

When the server is having issues, so the served front page is basically a cached static HTML file. The anti-bot limiter applies during normal operation. These are not the same thing.

---

On Feb 28, 2023:

> Hi sph

> Yes, you ran into HN's anti-bot software, which blocked your IP. Sorry! Our intention is definitely not to block legit users, but we have to be somewhat aggressive about detecting bots because we don't have much performance to spare.

> I've unblocked your IP. If you browse HN while logged in, that should immunize you against this happening again. Also, if it ever does happen again, you can unban your IP as described at https://news.ycombinator.com/newsfaq.html. But you have to do that from a different IP address, of course.

> Let us know if you have any further trouble,

> Daniel (dang)

Emphasis mine.

That's all well and good, but in no way demonstrates that the OP you replied to was wrong about logging out having been suggested as a fix for slow site performance.

I was replying to a user saying it's preferable to stay logged off at all times except when needing to post or comment.

User-accounts mean limits can be applied per-user instead of per-IP. Especially when a lot of us are connecting from the same office/site, or the same corporate VPN.

But being logged out is more efficient for cached responses. The more load the site is under, the more this pays off.

It makes sense that they're both true - it's not a case of which is more recent, it's what problem they're solving.

I’ll make sure and open HN in a second session logged-out to help when things are getting slow

I can also confirm that logging out (incognito tbh) fixed the errors for me in the past.

Backend upgrades I assume. The fact that this even exists, is free, is frequented, and has super valuable commentary is enough to make me wait. I get way more value out of HN than any other site I visit. It’s not my most frequented but it’s my most valued. Try not to compare it with a SaaS you pay for or that sells your information to pay for it.

Isn’t the general ethos here that HN doesn’t care about some number of 9’s of reliability? They just upgrade stuff when necessary and probably bang on prod a lot more than big corporate sites. HN is a nice to have, and keeping it that way allows it to remain administered by a small crew.

Dan gave some traffic numbers in 2015, and the site has grown a lot since then:

"Roughly 2.6M views a day, 300K daily uniques, 3 to 3.5M monthly uniques. It depends on how you count, of course."

https://news.ycombinator.com/item?id=9220098

> what happens, why do we get that message

This message is a HTTP 500 error page. 5xx are backend unreachability-related, so likely "the app" dies and has a long-ish boot time, or start is delayed by waiting for cron event.

Not doubting the experiences of anyone else, but I have only been a daily user of this site for the past 18 months, but not once has it been down for me in the morning or evening when I reach for a dopamine hit.

Maybe something to do with the cut cable and traffic routing?

https://news.ycombinator.com/item?id=39506931

Speculation – but Google moved HN up in search recently and I suspect that's resulted in a spike in traffic. I assume the downtime is capacity related.

I'm getting the same but only when using VPN. Not sure if there's a connection or just a correlation with me using the VPN during the day.

I'm not using VPN

it's to remind you that there is no place like hn else where.

also to remind you that one does not need to check HN when you have 5 minutes to kill. Go take a breath and have a walk.

You might be clicking on a bunch of links at once (triggering some anti-ddos/spam measure?). Happened to me once or twice. I assumed it might have been because my dynamic ip had been "used by robots" prior.

Is there any open metrics on HN hits / usage etc? Curious on growth of this platform. For me is my #1 place I regularly go to each day. Other sites less important to hit frequently.

Go easy. The new hamster is taking a while to train up.

I was always under the impression it was a sort of shadowquasiban/shadowratelimit mechanic

it definitely shows up a lot more via vpn

Yeah, it’s been down for me somewhat regularly in recent months but I find the outages charming.

Same here in the last few days between 10 AM and 1 PM (CET)

I have never seen that message before. Also from Europe.

HN is my only way to see if something is down

Motivation for you to get back to work.

> ” Did you post something negative about Google/DEI? Or maybe just upvote it? Or skim one of those threads. There you go. Time out for you." <---- this is particularly disturbing IF TRUE”

This is bullshit. I am sorry you decided to edit your comment to include that paranoia. I am seeing these errors a lot and a lot of times it’s when I just get to the site without any interaction.

I usually comment very uncontroversial things and get the error all the same.

Ps: I don’t know what’s “DEI”

does this bother you or you are asking out of technical curiosity ?

Considering how often it is, it is really bothering now, because it's wasting my time.

I don't know if you are being clever or funny

To or or not to or, that is the question here.

I can’t believe that this elite community of techies are chiming in with the equivalent of “works on my machine” :-)

It's very weird how most websites already figured out how to keep their services up under load, but a CRUD text-only forum, with an "elite community of techies", can't.

I would be curious to know what is the big challenge here.

Probably the fact that the software was written decades ago, hacked together and also the author decided to first write their own lisp, then implement HN software with this lisp (called Arc).

I'm guessing that has something to do with the difficulty of changing it.

I would wager that it has more to do with the fact that it's served by a single one-core server: https://news.ycombinator.com/item?id=39548439

If anything, that showcases that the software stack is quite capable, however old and hacky it may be.

And I suppose that HN is not too high up on the list of priorities for YC :) It works most of the time, and dang can put out any fires...

If it can't run on multiple cores, and a single core isn't enough to run it well, then I would say that the software stack is not quite capable.

Racket, the language Arc is built on top of, can handle concurrency and parallelism, so it is possible. If someone has time/energy, that I don't know.

However, Arc was initially written in sometime early in 2000s. I don't think dual-core processors became more common at home until later...

I almost wonder whether they even care about changing it, because why? This site has been completely functional aside from some load-bearing issues here and there. It's not like it's an ad plagued, high stakes, profitable venture.

I agree, I would say it's perfectly capable just the target for success is different. It's not a commercial venture aiming for 99% availability and a dropped request here and there isn't that impactful.

I'm embarrassed for them.

Used to happen whenever you criticized communist China so I assumed it wad an anti-naughty-posting feature. (Thankfully there’s been some relaxation on this, as evidenced by this post being visible.)

Might be worth considering moving the arch of the site to a SaaS solution like netlify to really benefit from the scaling capabilities of serverless.

Haven’t the serverless horror stories scared you already?

I hope this is a joke.

It's NEVER down for me. I've been visiting for a few years now. Multiple countries, ISP's always up. Maybe it's your ISP censoring?

Have a look at these

https://github.com/tg12/RUBC

https://en.wikipedia.org/wiki/HTTP_451

You're visiting too infrequently.