Hetzner removes traffic limitation for dedicated servers
wiki.hetzner.de
wiki.hetzner.de
Many years ago: HDDs died like I've never seen it before, one every few months with just a couple of servers.
ca. late 2013: We had some production systems running on Hetzner hardware, we regularly! called their support to tell them the DC lost connection because our monitoring apparently was better than theirs and they acted surprised for not hearing something yet.
Only like 2-3 years ago: Got a new server, had SMART issues and had to have the disk replaced before the first 24h after ordering were up.
An ex-coworker of mine joked they must have added the "please change the disk" option in the support form just for him because he was running 20+ servers and actually had to call them every week to replace a dying disk.
Yes, this was all years ago - but I'd rather pay servers out of my own pocket than be oncall for production servers there, ever again.
As long as the health of the HDD is 100% with no bad sectors there's no reason not to reuse it.
When you get your new hetzner server check it with smartctl, hdsentinel etc, if any show bad sectors or health other than 100% I am sure they'll be happy to replace it for you.
At work we have some Hetzner servers. About two times a disk broke; with RAID it was not a problem, a hassle, at least the first time, it was, yes.
Looking at the recycling issue I wonder if you are using so many disks (that fail) if it is possible to ship them to a low wage (sorry) country have them taken apart and rebuilt using parts from other disks where different parts failed?
You are (from my 10+ year experience with Hetzner at least).
:)
i ve also had some disk problems 5+ years ago, but not recently, even with the cheap bidding servers. Nothing that wasn't justified by their way-cheaper-than-others prices
I had a production server with two HDDs in some form of redundant raid configuration and both died at the same time. You somewhat lose the benefits of both drives mirroring the same data if both fail!
At one of my previous workplaces, we had a park of around 3k servers, with 2k of them on Hetzner. At such scale, you can not physically escape from the fact that all kinds of weird stuff happens with hardware, like buggy raid cards, physically unrebootable servers, dead cores, faulty and fake memory, bitrot, throttling due to underpowered PSUs,etc.
My favourite find was a "xeon 2637v3" with 6 cores (2 extra cores for free! unlike the normal sku.)
And we always had argument about their service, availability, giving us three faulty raid cards in a row, sued them few times for a refund, but... the company still liked them sooo much.
Also I wasn't really complaining I'd like to think - I was just stating facts. I saw like 10 "this is so awesome" replies here, am I not allowed to say I've not been a happy customer?
How is it possible that Hetzner (and other unmetered hosting) are giving it out for free? If you max out the 1 Gbit for a month that's ~300 TB.
GCP and AWS would've charged you $30,000 for that!!! Are the cloud providers just not competing with each other on price? How has the price remained so high for so long?
That said, I've frequently seen parallels between razor + razor blades pricing and how AWS/GCP price compute vs bandwidth. Computing is cheap, by some measures, but almost all customers will egress that data somewhere and once you've done your computation, you'll pony up whatever prices they demand for letting that data "egress"
You mean the big three. Other cloud providers are certainly competing: Digital Ocean is $0.01/GB, Oracle Cloud $0.0085/GB, OVH and Scaleway are unmetered.
That said many apps can deal with more latency. Image sharing sites would care more about cost vs latency compared to a multiplayer game.
GCE introduced a tier of lower perf but cheaper network. Hopefully the trend continues.
AWS and GCP both have much more investment into their own backbone, plus higher quality transit providers.
Second answer, because nowhere near all of the customers will use all of the capacity all the time. Economics of ISP oversubscription.
Also their new cloud options are top notch.
Discovering the robot marketplace[0] (thanks to HN) was an eye opening experience for me that showed me dedicated servers were viable and could be competitive with the VPSes I'd been purchasing/using for years (I was so excited I wrote blog posts about it). I've recently been thinking about making a subreddit for hetzner (r/hetzner), so fellow users can talk with each other but I am worried it would do more harm than good (increasing the burden on them in terms of having to do PR in yet-another part of the internet that can be pretty toxic in terms of community at times).
You can even deploy to them with terraform[1].
-- warning, rant/ramblings below --
I feel like I say the same thing all the time, but I'll repeat it again here -- if devops/infrastructure goes at all correctly in my mind, going to one vendor for both your compute/data and your value-added services (so asking AWS for EC2 machines and to run RDS on top of them for you and offer support) is going to dissipate.
Smaller companies can run sufficiently niche (many might disagree but I think running postgres is within this ) with more innovation and decent support for less cost than Amazon can. The vast majority of people who run RDS with something like Postgres as a backend are unlikely to run into crazy issues simply due to how well postgres is built, and how well known it is, outside of gross misconfiguration.
Companies are already offering on-your-cloud solutions that use EC2 machines to run their own software and offer support on top -- as container orchestration systems become even more widespread I think people will stop asking AWS or GCP for more than basic compute/data and a few tools they're really good at, and go to more specialized smaller vendors.
[0]: https://robot.your-server.de/order/market
[1]: https://www.terraform.io/docs/providers/hcloud/r/server.html
If you really want to penny-pinch unbundled solutions that charge you separately for compute, storage and ip addresses might be even cheaper, e.g. on scaleway you could get way with 1€/month if you go ipv6-only on their smallest instance type with a minimal amount of storage. Plus you get unlimited traffic with them.
I'm really digging their baremetal and ARM offerings -- both C2M and C2L look great to me and no setup costs is even better.
While I don't know if it's worth the headache to try that hard to cost-maximize I love that it's an option! I'm personally OK with paying more for a bit more breathing room on the utilization side (good ops is to keep your servers @ no more than 70% in case of burst right?).
[0]: https://www.scaleway.com/pricing/#anchor_baremetal
[1]: https://www.terraform.io/docs/providers/scaleway/index.html
I'm working on a new startup around a personal web archive and annotation system (https://getpolarized.io/) which uses Firebase but I would love to figure out how to use Hetzner as their prices are amazing.
Aside: on the frontpage "forum" links to GitHub, and Discord links to Reddit.
Is it just prioritizing or are there licensing/copyright complications ?
Tried buying one on the auction once, went through a whole rigamarole to confirm my identity, send them ID pics, etc. In the end they decided to cancel the order due to "suspicious activity". I was baffled by that as we hadn't had any conversation besides sending them the order info & credentials. I assumed it was a weird false positive... wonder if I should try again sometime.
Elsewhere highly recommended, by the way:
I've always wondered how people run their own (personal use or side project) services on bare metal or VPS providers manage them without spending too much time. I've found that trying to run even a few services and have a reasonable solution for backups, software updates, etc to take more time and effort than it seems is worthwhile, it makes Heroku and App Engine particularly appealing as a lot of the issues are kinda solved with managed cloud providers. I would love to hear how people make this work.
Another investment into a couple python scripts and API access to Linode (vps provider) and you can easily auto scale on demand with the best of them.
As a practical example, I run a cluster of auto scaling servers for around $500/month. The same infrastructure and traffic cost me almost $3,500 on AWS.
Great to see them features on HN!
They are the best kept hosting secret.
I'm sorry that you've had a bad experience, but we can help. Our support team will gladly increase that limit for you on request.
On the stories that you read, I hear you. It's a tough situation because I don't want to discount anyone's story. What I do want to point out is that there are two sides to every story, and in such a relationship one party has the duty of protecting privacy of the other. Sometimes we make mistakes and we need to correct it. Sometimes what we do upsets people when we work to protect our customers from those who would seek to abuse our platform. Abuse of our platform lowers quality of service for everyone, and that is why it is our duty to manage that. Each story will have it's own variables, and I'm happy to discuss anyone's situation with them personally.
If you have any questions or there is anything I can personally help with, please feel free to reach out at jdonnell@digitalocean.com.
Jarland
This month I "purchased" another EX41 server. Something made me wonder if the hard disks on the machine allocated to me were old ones previously used for some other customer, so the first thing I did after installing Linux was to look at smartctl output. Not only were the disks new (power on hours) but as part of provisioning the machine, they seem to have performed a "long" self-test on the disk (see smartctl -t in its manpage). I was impressed by this, because they seem to have checked that I didn't get bad disks on arrival. This is probably expected of such a service provider.. but I've seen worse.
Quite satisfied with support too. (Though I wish I could get a hold of a /48 IPv6 subnet from them, pfSense doesn't like /64 only networks).
Please pester them about this. (Technically, you can, but they charge extra, which is just braindead.)
The basic gist is that pfSense wants to be allocated a /64 via prefix allocation via a router advertisement (SLAAC or DHCPv6). Only in this configuration can you set the LAN interface to track the WAN, which allows you to use a single /64.
Sadly there isn't a dedicated server provider out there that I know off that cleanly supports this configuration.
In almost all cases you'll have to manually assign subnets inside the /64 (ie a /65 for the WAN interface and a /65 for LAN) so you can route around traffic. It's a bit annoying to setup but possible.
(It's still much better than OVH's IPv6 support which is just plain broken)
I never really saw a reason not to do it.
When you order, you can specify extra things you will need. For example, you can ask for a secondary NIC on each server, and to connect up the secondary NICs through a dedicated switch. There are some caveats though:
* more expensive. When your servers have non-standard configurations, Hetzner charges a fixed extra per-server, per-month fee ("Flexipack"). The fees add up.
* The servers need to be in the same rack. If you order them all at once it's fine. But if you order, say, four servers now, and the fifth server a month later, the fifth one will go on a separate rack and won't get the private access to the four.
* Setup takes longer. If you order standard configuration, it will be ready in minutes/hours. When I ordered the extra NICs and switch, it took about a week to complete the order.
* A bit more configuration required. Primary NICs work out of the box. For the secondary NICs to work you need to tweak /etc/network/interfaces and the routing table.
Good experience, in spite of the circumstances.
And the remote hands at hetzner are good, especially if you know how to work with DCOPs. You know, have redundant systems, shutdown the broken system so they can just do whatever they have to do once they're around.
I moved pretty quick from there after that.
Because they're a discount provider. OVH has the same problem.
Organization,Sessions,Page Views,New Users,Event Count Amazon.com,6436,9307,3823,140 Microsoft Azure,5798,8988,4645,2 Google Cloud,3313,5425,2486,0 OVH SAS,1715,3349,872,0 TOPNET,1657,5133,1409,14 OVH Hosting,1628,2787,1232,7 Digital Ocean,1267,3050,816,5 Hetzner Online GmbH,988,2051,529,0 Rackspace Ltd.,926,1644,376,48 Linode,809,1008,272,4
For dedicated hosting there is https://robot.your-server.de/doc/webservice/en.html
I just wish they had NixOS images so I'd save a few minutes of initial deploy time.
I accept minor things like this as part of the price for hosting in Europe, and have certainly experienced the same issues going the other direction (i.e. me reaching US-based services) more often.
They have an API.
If you don't have an account you'll likely need ID verification, it was a while ago but to my recollection they'll simply hold the purchase for you while they verify so you don't loose a good spot. Plus you have to do it only once.
Also, Support takes much longer, maybe days, to respond.
If you see them as learning servers they are ok, plus you learn to negotiate with support ;)
TL:DR Its not as rosy or great as some people here suggested, and their response on contacting them came off as a little harsh to me.
Here is what truly happened, I registered, entered my personal details, confirmed my email, entered my card details, they then deducted about USD 0.010 from my card, which is okay, many cloud services do that to test the card validity or something like that.
So the next step is you have to submit a copy of your ID or passport, the reasons they give for this made perfect sense, so i sent in a photo of my ID, then in the morning, about 7 hours later, I got an email saying my account was rejected.
Easy, so I try to call their support team, to find out what i had submitted incorrectly, or what i had missed. The answer is not what you would expect given the praise their customer reps had on this thread, I was told the rejection is a management decision and i had to reapply with another email. note that even if i had wanted to reapply, now i couldn't because i wanted to know what data i had not included or submitted incorrectly.
I call in again and I am told the same thing. So i try to login and my account is deactivated, thats expected.
So that's my experience.
Sidenote: Do not start with a greeting like "good morning" when you call their support team, both of the people i called did not answer to that and waited for me to speak again, before they said anything, which i felt was quite embarrassing, like as though saying I should know better and go straight to the point when i call in.
Maybe someone else who got through has a different narrative, but if you also failed then your story is similar to mine. but If you want to have a great week, don't try them b'se they will steal your happiness for a day. Stay with AWS or GCP or Digital Ocean or whatever
0,125 * 86.400 (1 Day) = ~10,8 TB per day
Theoretically you can also use inbound and outbound at the same time.
I have lost my data twice on different hardware. The first one was totally Hetzner's fault since they have failed to identify the faulty disks when they check the disks for it. They were like `oops, sorry, it seems second disk was faulty too` AFTER I tried to re-build the RAID array on a recovery attempt. And the second one was being a bit unlucky, the RAID card didn't detect the faulty disk and tried to sync the broken disk with the healthy ones. So I lost data on a RAID-10 array, twice.
I asked them to do tests on the faulty hardware to identify the problem, to confirm my suspicion, so I could go back to my clients with a proper explanation and things got interested at this point;
While I was talking to the technical people at Hetzner and trying to explain why I think the RAID controller is the problem and how the RAID logs are confirming that, a guy called Arno, Head of Product Advice and Sales took my ticket over. He said;
``` First of all we want to apologize for what happened. Why it happened is hard to say. As you wrote at least one disk was so faulty that your data was corrupt.
Those things can happen and from our point of view no one can prevent from happening again. The only things that help are good backups and a good recovery strategy, which I think you have both.
Of course extra efford on your end occurs for recovery and since it happend the 2nd time, we want to compensate you for that.
What compensation would you expect from us? ```
which was kind of ridiculous because I didn't ask for a compensation, I didn't even imply that I want a compensation for what happened. I was simply requesting a solid explanation and/or confirmation and this guy was saying that I lost my data on a RAID-10 array because 1 (ONE) of the disks were faulty.
I asked him to handover the tickets to the technical people since he's from sales and what he says doesn't make sense. Also provided him some numbers about how much money the sites were making on that lost server, just to prove how wrong he is to approach to the matter from that perspective and guess that was the bit what triggered him.
He fought so hard to not handover the ticket back to the technical people, explained how much he knows about RAID-10, how loosing the server is our fault while making so much money the sites because the server was cheap (WTF?), they won't do the testing because they host many more servers with the same hardware and they are just working fine etc.
After using Hetzner for 7 years, I cancelled the service I take from them, along with other 11 company accounts and tens of dedicated servers and never looked back.
Hetzner is a company that ask you to move from their network to some other provider when you ask what happens if you get DDOSed (this happened before they introduced the DDOS protection).
So never again...
Is it just because you don't like to see something negative about the provider or is it something else?
Hetzner provides unmanaged dedicated servers, thus you are the one responsible monitoring the hardware. The only thing Hetzner does is swapping HDDs you identify as faulty (you provide them the S/N of the faulty disk via the support form).
Personally I rent the super cheap Serverbörse servers for personal use. The HDDs in there are fairly old (as it's their older generation servers). Over the past 6 years I needed Hetzner to swap about 5 HDDs. When my server logged issues when accessing a disk, I requested a swap, they swapped it less than 30 minutes later and I started the rebuild of my RAID. All without issues.
> The first one was totally Hetzner's fault since they have failed to identify the faulty disks when they check the disks for it.
Did you mean to say that they removed an incorrect disk when you requested a swap? Or did you expect them to monitor the hardware for you?
I totally agree but working on a remote machine it's not always possible, even through KVMs, so Hetzner had to step in on this one.
There were 2 faulty disks from the same 4-disk Raid but once they have found the first, they assumed that first disk was the issue and didn't test the rest, so that ended up with loosing everything.
Even if they are helping as a courtesy, they should test all the disks and do this right before let the customer continue.
> but once they have found the first
How did they find the HDD? As I said: They don't monitor your server. Did you request replacement of the HDD? If you request replacement of one disk, they swap one disk.
> Even if they are helping as a courtesy, they should test all the disks and do this right, before the customer continue.
How should that work? They remove all the HDDs from the server, put them into a different computer to run a test? That would be a privacy nightmare and add unnecessary downtime. I keep track of my HDD status (smartmontools, mdadm Events, ZFS Event Daemon). I know exactly what HDDs I want replaced. They just perform the actual, physical, replacement for me. And that is absolutely fine.
1 - I woke up to an unresponsive server in the morning. Couldn't reach to the server in any way.
2 - Got in touch with the support and they asked me to give them like 50 mins to check the hardware and disks and get back to me with;
Dear Client,
We have found this disk S/N xxx defective. Please confirm the full data loss on it in order to replace it.
right after that they wrote;
Dear Client,
We have found this disk S/N xxx defective. Please confirm the full data loss on it in order to replace it.
I asked if they mean data lost on this single disk or the whole array, they said;
Dear Client,
We cannot tell you how is going to react the raid controller as can happens also data loss if it is defective. For that, we recommend before to take Backups of the data. When you are sure to replace the disk tell us, confirm data loss and we will do it.
after many emails and sharing error messages I've got from the raid controller Hetzner said;
Dear Client,
We see our test reported that 2 drives are faulty, but we have replaced one of this only. Now, we could offer to replace the second faulty one (SN: xxx).
Could we do this anytime?
I don't know what kind of test they did but I believe the report he mentioned is from the very first diagnose they did.
I wouldn't know the details since I didn't argue on how they could oversee this, I tried to find a solution.
Edit : I fixed the order of messages.
It should have detected the disks going bad and allow you to request a replacement with 3 of 4 disks being good and then later a replacement of the second bad disk with 3 of 4 being good, no?
> I asked if they mean data lost on this single disk or the whole array, they said;
IMO this is the correct answer from Hetzner. They cannot know whether you run RAID 1 (which would be good), RAID 5 (which would have killed the array), RAID 6 (good) or RAID 10 (might or might not be good). Asking before replacing could possibly allow you to save data in case the disk is not completely dead, but rather has a bunch of faulty sectors only.
The only issue I personally see is the bad English. I am German like Hetzner and even I have issues understanding that grammar.
[1] https://www.golem.de/news/hetzner-und-netzneutralitaet-extra... (German)
The same as Level3 charges more than others (even though not as extreme as DTAG) because they think their pipes are premium.
I mean, yeah, sure, it sucks for the customer. What I don't understand is your implication that this is Hetzner's fault. Both sides could accept the other side's conditions, neither side did, all of that is "just business".
What isn't "just business" is asking much higher prices than the competition for the same kind of service and expecting customers to buy it. And if you find that there are customers who do buy anyway, that's a hint that you are dealing with a monopoly situation, which is appropriately described as "holding people hostage".
2. Assuming you don't really mean paying for transit, but paying DTAG for accepting traffic that DTAG's paying customers have requested to be sent to them from Hetzner:
2.1 Why?
2.2 When you say "they should pay", what you are really saying is "all of Hetzner's customers should pay". Thanks, but I am not interested. I intentionally don't use DTAG as a customer, and I am not interested in subsidizing those who do.
2.1. DTAG has a market share of around 40% in Germany. Without a direct connection to them you won't reach them properly. The connection trough other Tier 1 providers is is awful congested.
2.2. I don't want to sound offensive, but it doesn't really matter what you are doing. You are the exception, most people and your customers don't care about their business tactics, often times it's their only choice to go with DTAG.
Of course the traffic pricing would increase, this is why I called it absurd to wave the traffic pricing, because Hetzner is offering a crippled line to 40% of Germany.
If the connection between DTAG's network and Heztner's isn't good, shouldn't both networks work to improve the situation?
Just because DTAG also offers transit, doesn't make a peering connection a transit connection, so, no, it doesn't. That's as absurd as saying that DTAG is also very good at LTE networks, so technically that makes an internet peering connection an LTE connection. All this is about DTAG accepting traffic destined for their customers. Noone cares what DTAG charges for transit, and noone minds if DTAG drops transit traffic at peering points.
> 2.1. DTAG has a market share of around 40% in Germany. Without a direct connection to them you won't reach them properly. The connection trough other Tier 1 providers is is awful congested.
Which only explains why you would want to have a good connection to DTAG if possible. I asked why you should pay massively above-market rates for that.
Also, suppose DTAG asked 1000 EUR per TB of inbound traffic. Would you still use that argument? If not, why not?
> 2.2. I don't want to sound offensive, but it doesn't really matter what you are doing. You are the exception, most people and your customers don't care about their business tactics, often times it's their only choice to go with DTAG.
It doesn't matter what I am doing when as a matter of fact I am doing what apparently most Hetzner customers are doing (that is: not buy DTAG peering traffic), which presumably is the exact reason why Hetzner still doesn't include DTAG peering traffic in their default package?
Or are you saying that you think it shouldn't matter because Hetzner should raise prices for everyone, including me, to pay for DTAG peering traffic that most of us don't care about, because you know better what I should care about?
My server has essentially no DTAG traffic, and certainly no bandwidth or latency critical DTAG traffic, so if you think that I should care about DTAG traffic, you are just wrong. Not every server is a public web server. Not every public web server is targeted primarily at a German audience, or even a German audience at all. I know when I need DTAG traffic and when I don't, and I don't care to subsidize DTAG traffic when I don't need it.
Also, I don't think there is anywhere in Germany where DTAG IP is your only choice.
> Of course the traffic pricing would increase, this is why I called it absurd to wave the traffic pricing, because Hetzner is offering a crippled line to 40% of Germany.
That's just bullshit. Hetzner is offering a completely uncrippled connection to all of Germany. It's just that DTAG for some reason inserts a bottleneck into the connection of their customers instead of letting through the packets that Hetzner delivers to them completely free of any congestion.
All you are saying is that Hetzner should raise prices so that I have to subsidize a protection racket that you choose to be affected by instead of giving me the product that I actually want.
And all that while there are plenty of options on the market for high-priced hosting solutions that do include DTAG peering traffic, so it's not even that you can't buy what you want to buy. Just rent a Strato server, they are a DTAG company, and you will have no problem sending traffic to DTAG customers, instead of demanding that Hetzner offer the same product as Strato instead of what Hetzner's customers want.