Tell HN: The Hetzner Experience - Invisible Outages | Hacker News Reader

Tell HN: The Hetzner Experience - Invisible Outages

40 pointsAmazingTurtle12 comments

I'm a DevOps engineer at Schäfer Shop GmbH, and we've been running into recurring quirks with Hetzner's cloud infrastructure. Occasionally, volumes randomly disappear or become unavailable, with no indication on Hetzner's status page. Support ticket response times vary wildly - from hours to an entire business day.

Yesterday, we had a particularly stressful incident involving Hetzner load balancers in Falkenstein. Our Kubernetes control planes were unreachable due to load balancer targets showing as unhealthy. We quickly worked around the issue by deploying an identical load balancer configuration in another region. Despite explicitly instructing Hetzner support not to recreate our resources (since they're managed via Terraform), they manually recreated the load balancer anyway, causing momentary panic - though thankfully our Terraform state wasn't impacted.

We pay nearly €20,000 per month for Hetzner's services, yet they refuse to offer a direct support hotline, even if we were willing to pay extra for it. What's especially troubling is their persistent silence on these outages. Hetzner's status page showed no signs of this incident, neither during nor after. This pattern makes us question the transparency and purpose of the status page itself.

Have any of you experienced similar invisible outages with Hetzner?

6 years ago when I used Hetzner, it was widely known as unreliable, a provider where you could get hosting that delivered good performance for a cheap price. The tradeoff always was that you needed to treat each machine almost as EC2 Spot: it could go down at any time.

You also needed to consider that when this happened, the data inside the machine was mostly lost. Finally, you also needed to plan to graduate out of it as soon as you had enough money to go either to a colocated data center or the "real cloud".

I kept Hetzner as a backup provider in more than one company, mainly to have real machines for take home tests, back when hiring was plentiful. Even so, we often faced problems with the machines going down due to hardware or networking issues, and the need to rebuild them from the ground up. Those mirrored all tales of woe everyone in the department had from years of working with Hetzner, sometimes losing production data because the rules of the game were not followed.

So it seems that 6 years later their scale has increased but the experience remains the same. On the bright side, kudos to Hetzner for teaching waves of engineers about reliability and disaster recovery during all these years.

> What's especially troubling is their persistent silence on these outages.

Based on many years of experience, all providers are guilty of that. Only large scale outages or ones that just couldnt be ignored are reflected on status page. This doesn't necessarily mean malevolence on provider side - their sensors just may not be good enough to spot the issue.

On larger scale - why would you choose hetzner and then complain about uptime? Its a well know provider with low prices and low reliability. There are tons of businesses who find this model suitable for them. If yours is not one of them - just switch to something more reliable. Granted, your bill will likely be 2x+ of 20k eur, but you get what you whine about.

As old adage says, we can make this project fast, cheap and with amazing quality, but you can choose only 2 options.

We’ve used Hetzner dedicated servers for many years now, but not cloud.

Our experience has been excellent, but we also design for the platform. Redundant dedicated networking, multi-AZ, networking failover, RAID, k8s, Mayastor, etc.

The worst issues we see are occasional scheduled outages of an upstream router. This will take out an AZ for external traffic for about 20 mins, but the internal dedicated network will ensure internal services all stay up and quorate.

It’s not cheap, but it’s still cheaper than AWS.

I think their dedicated offering is probably more stable as it has been around longer, and it is also much much simpler. They need to provide networking, power, and finance the hardware. All of which is very much solved problems.

(We’re https://lithus.eu, if anyone is interested. You can contact me at adam@…. I’m on holiday this week, but back next week)

I have been a Hetzner Cloud customer with a handful of servers (on the lower end, all less than €150 / month) for a few years and have had no issues, other than one server, a couple of years ago, refusing to download packages from deno.land/x/. I think I was somehow rate-limited (I wasn't doing anything crazy, so it was surprising), but definitely not Hetzner's fault and it resolved after a couple of days. That being said, I also haven't used their volumes or block storage a lot.

We've had a very similar experience over the past year. We've been using Hetzner for over a decade and, until recently, we were really satisfied with their services. But in the last 12 months, the reliability has noticeably dropped.

Most of the issues we've faced are related to their Storage Boxes, multiple incidents where they were completely unavailable, sometimes for hours or even days. What’s frustrating is that these outages are never reflected on their status page, so you're left in the dark unless you open a ticket yourself. Even then, the only explanation we usually get is that the specific Storage Box is under "heavy load", and the suggested fix is always to migrate to another box. That might be fine for infrequent use, but it's not acceptable when you're relying on it.

To be fair, Hetzner has been a solid provider for many years, and we’re still hoping this is just a temporary rough patch. We really hope they get things back on track soon.

Could it be related to people in Europe suddenly looking for European alternatives? Like suddenly they got a lot of new customers in the last few months?

We have 100s of HCloud machines and never encountered similar issues, fortunately.

Just the typical server outages and a "Fault report" notification Email from Hetzner

I know Hetzner just from a small customer perspective, I think I never had anything else than their discount servers.

However, I don't run anything mission critical with them as they don't really have a reliable support. Just using their cheap dedis for background tasks.

The server I have right now is stable, but I had different experiences before as well. And their network is unreliable either way, timeouts etc...

I've had a lot of issues with resources located at Falkenstein, there seems to be issue at that particular location. I've moved to other locations, and so far it's running fine

I remember reading cases were a competitor had launched a very small DDoS attack against a service and Hetzner null-routed the IPs effectively taking the entire service offline.

Support was non-existent so nothing has changed there.

They also have an extremely strict T&Cs that hold customers liable for user uploaded content.

Crazy to see this now as we also had a very similar issue yesterday! All backend nodes of our kubernetes cluster were suddenly and inexplicably showing as unhealthy despite being all green on the cluster side, no signs of issues whatsoever.

In my first job we've used Hetzner for Data Science work. The we lost servers twice in a year back then.

Reply on news.ycombinator.com