Hacker News Reader

Comment by bhouston | Hacker News Reader

> Until this change, customers who have used fewer resources have covered the costs, in a way, for other customers who have used much more resources. We want to make things more balanced.

I may know one of the culprits -- whom I will leave unnamed here. But the company, who is fairly popular, built out their own CDN via putting a bunch of nginx caching proxies on various Hetzner servers around the world. It apparently was really cheap and very effective. Given that they were bootstrapped and this was prior to Cloudflare really being that popular, it was a great strategy. This was true like 8 years ago, so maybe it has changed in the meantime.

View on HN

Sounds like a completely legitimate use though. Hetzner were widely telling people about that 20TB limit, so why would they be surprised when people use them as CDN boxes?

Where's the surprise? It's the classic business 2-step - drum up interest with "too good to be true" features, then cut them back. The marginal customers who need those features leave (and are too expensive to keep), everyone else is used to your product and stays.

Hetzner has been in this business for a very long time and I’m sure half the German small to medium sized companies use Hetzner. They have costs as well and with virtual servers they just assumed the usage pattern was similar to their eu data center and that didn’t work out so they increase the prices a bit. So that on average their calculation works out again.

Like, this is not Netflix tripling the prices over a few years.

That or they just didn't dimension well their prices and now it's biting them back.

I prefer to assume naivety over malice/negligence.

MBA value extraction 101.

Until you remember that marketing is a separate department from finance which is a separate department from ops/engineering.

The engineers said 20TB in aggregate was fine but likely didn’t consider the “bad apples”. Marketing obviously wants to use the biggest numbers and then finance comes in with the hammer and dev points to egress as an simple way to upset rhe fewest number of real customers.

It does make sense: The average was low enough that a 20TB cap worked. Then marketing started boasting about the 20TB limit and attracted a bunch of high-bandwith customers, thereby driving up the average making 20TB decidedly "not fine".

As an engineer, if you don't qualify whether your answer is average or max, you've messed up.

I'm confident the marketing folks would just say something akin to "shut up nerd", cash their bonuses, and leave you with the problem.

As an engineer, NEVER give out averages without checking that they are consistent with the distribution of the actual data.

Averages tend to skew once exposed to adverse selection.

Don't think you find much engineers in the marketing department though.

I think culprits is a poor choice of words since it means someone suspected of a misdeed. I could perhaps understand using it for example for someone that tried to store a petrabyte of storage on a consumer unlimited storage plan. But in this case Hetzner set a specific data usage amount you are paying for so using that amount is not a misdeed.

> I could perhaps understand using it for example for someone that tried to store a petrabyte of storage on a consumer unlimited storage plan.

I couldn't, don't call it unlimited if it's not unlimited, using what you paid for is not a "misdeed".

Well, I doubt an extra $1/TB is going to stop Hetzner from being very price effective.

Then why does this only apply in the US? Are they saying EU customers are well behaved?

It also may depend on peering arrangements Hetzner has. If EU ISP more inclined to peer with Hetzner than US one bandwidth in EU will be cheaper for them.

- in EU there are large peering exchanges to swap traffic

- in USA no peering exchanges exists and you need to pay for your traffic most of the time. Few big operators in US and they enforce this.

Looks like some deal wasn't renewed and they lost a big chunk of cheap pipe or/and some of their upstream providers decided to do something with routing.

Also, Hetzner is way bigger in the EU than in the US. Good access to services hosted by Hetzner is thus more relevant to EU ISPs, because customers in the EU will probably use more services hosted on Hetzner infra. This gives Hetzner more leverage in the EU to negotiate beneficial conditions with regard to its uplinks and peering agreements.

Hetzner colo's for their US servers/locations. Perhaps their US bandwidth rates are much higher than their own DCs?

probably streaming platforms have better content on the US and everyone wants to exit there? so they are mostly serving US traffic for several vpns all over the world connecting to CDNs in the US.

Isn't Hetzner impacted in Europe at the moment from a cable cut? Wouldn't surprise me if there is a wave of people moving stuff over to the US because of that. I don't know much about cloud though. I believe "The Ship has arrived and repairs are underway, which will still take some time." and "the repair may take up to two weeks.".

Hetzner did pay somewhere around 20% of the cost of C-Lion1, so it wouldn't be surprising if it has hurt hem.

Last I heard from Cinia was that the cable should be fixed by the end of November, so sometime within the next 55 hours.

Seems done https://www.cinia.fi/en/news/cinias-c-lion1-submarine-cable-...

8 years ago, hetzner had no cloud offerings

The dedicated servers still have 20 tb traffic included

Do they have dedicated servers in US?

In Hillsboro, OR they been building up their data centers but this hasn’t converted to dedicated servers in the US yet

Do you have a source for this? My last info was they rented space in an existing data center.

What? Afaik they are unmetered. And real unmetered - there are people who are saturating their connections 24/7 while using them.

“Culprits” because they used the service they paid for within its advertised limits?

It’s the same with cloud storage providers. First give out a massive amount of storage and rapidly gain users, then cut it down after blaming people for “abusing” it. How about you advertise your correct capacity to begin with?

They are simply deflecting blame for their own enshittification.

>“Culprits” because they used the service they paid for within its advertised limits?

"Culprits" because it was their (legal) use of the service that made Hetzner rethink and change their service plan.

So then the culprits are the company’s own engineering and marketing departments for not correctly anticipating user demand.

Sure .. also them.

Having said that, I am usually empathetic to these kinds of 'unlimited' deals because even though they aren't really 'unlimited', they do tend to be generous to the average use-case and average user .. Inevitably, and unfortunately, someone decides to test the limits and the entire thing collapses.

It reminds me of the Blockbuster "No more late fees" policy, which was a really good customer-friendly policy (speaking as someone who regularly returned rentals late) .. but then they were sued because an aspect of the policy had Blockbuster charging the cost of the rental to the customer if it wasn't returned in some period of time .. and because that charge looked like a 'late fee' they got sued. Urgh.

Looked like? Sounds like it was a late fee. Look at us, we've got the cheapest plane tickets by $25 ($50 booking fee applies).

IIRC it wasn't a late fee it was a "you kept it over a month so you have to buy it price."

It’s not any more possible to correctly anticipate all pricing structure vulnerabilities, than it is to correctly anticipate all program and API security vulnerabilities. There is always a statistical chance of novel outcomes when humans are involved.

Or you take your best shot and then adjust as needed.

Want to give a clue on who the culprit is?

I was curious and checked if they are still using Hetzner. It appears not, so I can share who it was. It was https://artstation.com. Basically heavily oriented towards serving static images, so the CDN could have been really expensive. Doing a reverse IP lookup on cdn.artstation.com servers now resolves to Cloudflare and it has cloudflare headers on the response.

Reply on news.ycombinator.com