I have lost my data twice on different hardware. The first one was totally Hetzner's fault since they have failed to identify the faulty disks when they check the disks for it. They were like `oops, sorry, it seems second disk was faulty too` AFTER I tried to re-build the RAID array on a recovery attempt. And the second one was being a bit unlucky, the RAID card didn't detect the faulty disk and tried to sync the broken disk with the healthy ones. So I lost data on a RAID-10 array, twice.
I asked them to do tests on the faulty hardware to identify the problem, to confirm my suspicion, so I could go back to my clients with a proper explanation and things got interested at this point;
While I was talking to the technical people at Hetzner and trying to explain why I think the RAID controller is the problem and how the RAID logs are confirming that, a guy called Arno, Head of Product Advice and Sales took my ticket over. He said;
``` First of all we want to apologize for what happened. Why it happened is hard to say. As you wrote at least one disk was so faulty that your data was corrupt.
Those things can happen and from our point of view no one can prevent from happening again. The only things that help are good backups and a good recovery strategy, which I think you have both.
Of course extra efford on your end occurs for recovery and since it happend the 2nd time, we want to compensate you for that.
What compensation would you expect from us? ```
which was kind of ridiculous because I didn't ask for a compensation, I didn't even imply that I want a compensation for what happened. I was simply requesting a solid explanation and/or confirmation and this guy was saying that I lost my data on a RAID-10 array because 1 (ONE) of the disks were faulty.
I asked him to handover the tickets to the technical people since he's from sales and what he says doesn't make sense. Also provided him some numbers about how much money the sites were making on that lost server, just to prove how wrong he is to approach to the matter from that perspective and guess that was the bit what triggered him.
He fought so hard to not handover the ticket back to the technical people, explained how much he knows about RAID-10, how loosing the server is our fault while making so much money the sites because the server was cheap (WTF?), they won't do the testing because they host many more servers with the same hardware and they are just working fine etc.
After using Hetzner for 7 years, I cancelled the service I take from them, along with other 11 company accounts and tens of dedicated servers and never looked back.
Hetzner is a company that ask you to move from their network to some other provider when you ask what happens if you get DDOSed (this happened before they introduced the DDOS protection).
So never again...