Hacker News Reader

Comment by Mo3 | Hacker News Reader

Not only that. It's single-DC, and Hetzner is somewhat known for having serious DC-wide network failures from time to time.

The hardware used is consumer hardware too.

This is literally the opposite of what I would expect for a SaaS with uptime requirement... I feel like OP/the developers behind this might have made a really terrible choice and could be paying more in manpower than the savings they have by using a budget server discounter

Don't get me wrong, I have nothing against Hetzner, but this seems like an unfit usecase. This is literally what you use cloud platforms for..

View on HN

I run a one-person load-balanced SaaS on Hetzner. In the 2 years since I started I've had no customer-facing outages.

Over the same period, I think there's been two outages of AWS us-east-1 and a multi-region GCP outage. Plus weekly outages of a grownup unicorn SaaS like github or slack.

It's a tradeoff: the simplicity of such a setup reduces outage risk, and the costs saved can be spent on monitoring tools or extra redundancy.

Consumer hardware is totally fine if you use multiple machines - the probability of them failing at the exact same time is very low.

His stack already uses multiple appservers (so the loss of one wouldn't even be noticeable) and has a manual way of failing over to a DB replica.

It's often cheaper to work around node failures (which you often need to do for other reasons anyway - DC-wide outage, etc) than have more reliable nodes.

It's not single-DC (since they have multiple locations and multiple datacenters at those locations), but they are a single provider. However AWS is also a single provider. And as Facebook and large ISPs have demonstrated recently-ish, having a global footprint does not mean you cannot break everything at the same time in interesting ways anyway. Only way around this is to go with multiple vendors, but now whatever you build to use them at the same time/fail over between them will be the weakest link.

Hetzner also offers server-grade hardware btw. But if your setup is fully redundant anyway it doesn't really matter anymore if it's consumer grade or not (which the setup described in the article is).

We are discussing the service described in the article linked to with this post, which

1) runs single-DC 2) uses consumer hardware

I agree on the notion that with redundancy it is not as important what kind of hardware is used, but it still is something to consider (seeing how this is a one-person venture and server outages are still a huge time-sink), and I wouldn't even necessarily speak about true redundancy in this case

The article explicitly said the servers are scattered across the various datacenters at the Falkenstein location - ergo not single-dc. Now you can argue about the amount of shared fate, but they are separate buildings and hetzner lists them as separate datacenters [0].

[0] https://docs.hetzner.com/general/others/data-centers-and-con...

Lol... okay, point taken, single-location.

Not going to argue about the shared fate, as I've seen more outages than not affecting whole locations..

im curious as to why they dont have US West, Central, Asia data centers. Even with a US West, Central offering they could really do a number with their pricing discounts.

even better if they offer Singapore & Tokyo.

Hetzner has built a DC in the US East where they currently offer their cloud products:

https://www.hetzner.com/cloud

> uses consumer hardware

I run dedicated servers at Hetzner and the consumer parts are CPU and motherboard. Memory is ECC, NVMe SSDs are data center grade. How often do you see CPU failure?

I didn't know that thanks for sharing. Has there been major cases recently?

really puzzling that a healthcheck provider itself is not accounting for this potential issue.

Is there some sort of edgy reverse proxy here? Doesn't seem like it. Even then there would be a big latency.

https://status.hetzner.com/

We haven't been using them for anything in production, so I am not 100% sure - they have a backbone outage right now however..

There are no edge / reverse proxies being described in the OP article, just a HAProxy configuration within the same DC.

interesting....im also concerned that with the recent geopolitical developments, Hetzner could already be at crosshairs

In case you are talking about potential DDoS attacks, yes, they have been hit hard by massive attacks the last few weeks.

They also do not have adequate protection systems in place

About a month ago Hetzner was hit by an UDP DDOS, and as a mitigation they throttled UDP traffic on specific port ranges.

This caused major problems for Healthchecks, as I'm using Wireguard for private networking, and Wireguard uses UDP. Here's a postmortem about this incident: https://status.healthchecks.io/en/incidents/dcbcDNd89LptHoWi...

Reply on news.ycombinator.com