Hacker News Reader

Comment by breadislove | Hacker News Reader

Hetzner is really great until you try to scale with them. We started building our service on top of Hetzner and had couple 100s of VMs running and during peak time we had to scale them to over 1000 VMs. And here couple of problems started, you get pretty often IPs which are black listed, so if you try to connect to services hosted by Google, AWS like S3 etc. you can't reach them. Also at one point there were no VMs available anymore in our region, which caused a lot of issues.

But in general if you don't need to scale crazy Hetzner is amazing, we still have a lot of stuff running on Hetzner but fan out to other services when we need to scale.

View on HN

> Also at one point there were no VMs available anymore in our region, which caused a lot of issues.

I'm not sure if this is a difference between other clouds, at least a few years ago this was a weekly or even daily problem in GCP; my experience is if you request hundreds of VMs rapidly during peak hours, all the clouds struggle.

Right now, we can’t request even a single (1) non-beefy non-GPU VM in us-east on Azure. That’s been going on for over a month now, and that’s after being a customer for 2 years :(

We launch 30k+ VMs a day on GCP, regularly launching hundreds at a time when scheduled jobs are running. That’s one of the most stable aspects of our operation - in the last 5 years I’ve never seen GCP “struggle” with that except during major outages.

At the scale of providers like AWS and even the smaller GCP, “hundreds of VMs” is not a large amount.

If you’re deploying something like 100 m5.4xlarge in us-east-1, sure, AWS’s capacity seems infinite. Once you get into high memory instances, GPU instances, less popular regions etc, it drops off.

Now maybe after the AI demand and waves of purchases of systems appropriate for that things have improved, but it definitely wasn’t the case at the large scale employer I worked at in 2023 (my current employer is much smaller, so doesn’t have those needs, so I can’t comment)

Not a single VM possible to request on Azure us-east for over a month now though :-(

I was talking about real cloud providers :P

I don't use Azure much anymore, but I used to see this problem regularly on Azure too, especially in the more "niche" regions like UK South.

Note that we might be talking about two different things here: some of us use physical servers from Hetzner, which are crazy fast, and a great value. And some of us prefer virtual servers, which (IMHO) are not that revolutionary, even though still much less expensive than the competition.

The blocking of services on Hetzner and Scaleway by Microsoft is well known -

https://www.linkedin.com/posts/jeroen-jacobs-8209391_somethi...

I didn't know AWS and GCP also did it. Not surprised.

The problem is that European regulators do nothing about such anti-competitive dirty tricks. The big clouds hide behind "lots of spam coming from them", which is not true.

First comment on that post claims that according to Mimecast, 37% of EU-based spam originates from Hetzner and Digital Ocean. People have been asking for 3 days for a link to the source (I can't find it either).

On the other hand, someone linked a report from last year[0]:

> 72% of BEC attacks in Q2 2024 used free webmail domains; within those, 72.4% used Gmail. Roughly ~52% of all BEC messages were sent from Gmail accounts that quarter.

[0] https://docs.apwg.org/reports/apwg_trends_report_q2_2024.pdf

Worth noting that this seems to be about Hetzners cloud product, not the dedicated servers. The cloud product is relatively new, and most of the people who move to Hetzner do so because of the dedicated instances, not to use their cloud.

Hetzner's cloud offering is probably a decade old by now - I've been a very happy customer for 8 years.

Hetzner was founded in '97, so cloud offering could technically still be considered relatively new. :D

You're right! I seem to have mixed it with some other dedi provider that added "cloud" recently. Thanks for the correction!

My point of people moving to Hetzner for the dedicated instances rather than the cloud still remains though, at least in my bubble.

No problem (I'm just glad you didn't read it as snark)! I mean, even 8 years is relatively new compared to their dedicated box offering so technically you were still correct.

1000 VMs?

So you have approx 1MM concurrent customers? That's a big number. You should definitely be able to get preferred pricing from AWS at that scale.

We have extremely processing heavy jobs where user upload large collection of files (PDFs, audios, videos etc.) and expect to get fast processing.

I've ran into the IP deny list problem too, but for Windows VMs - you spin them up, only to realise that you can't get Windows Updates, can't reach the Powershell gallery etc.

And just deleting it and starting again is just going to give you the exact same IP again!

I ended up having to buy a dozen or so IPs until I found one that wasn't blocked, and then I could delete all the blocked ones.

This sound really intriguing, and I am really curious. What kind of service do you run where you need a 100s of VMs? Was there a reason for not going dedicated? Looking at their offering their biggest VM is (48 CPU, 192 GB RAM, 960 GB SSD). I can't even imagine using that much. Again, I'm really curious.

we have extremely processing heavy jobs where user upload large collection of files (audios, pdfs, videos etc.) and expect to get fast processing. its just that we need to fan out sometimes, since a lot of our users a sensitive to processing times.

I think they're great but it's unfortunate they don't have more locations which would at least enable you to spin VMs up in different locations during a shortage. If you rely on them it might be wise to have a second cloud provider that you can use in a pinch, there's many options.

We scaled to ~1100 bare metal servers with them and it worked perfectly.

Username checks out.

Blacklisted by whom?

AWS at least maintains IP lists of bots, active exploiters, ddos attackers, etc, that you can use to filter/rate limit traffic in WAF. Not so much AWS that blocks you but customers that decide to use these lists.

Ironic, given how often the attacks I spend time fending off are coming from AWS.

So AWS could list some IPs of competitors, just enough to make them look unreliable.

Nice IPs you've got there Hetzner, shame if...

The big cloud providers I am assuming

Reply on news.ycombinator.com