Comment by TimWolla | Hacker News Reader

Comment by TimWolla | Hacker News Reader

Parent Full thread

As the submitter of the link I cannot even downvote your comment, but something feels odd about it:

Hetzner provides unmanaged dedicated servers, thus you are the one responsible monitoring the hardware. The only thing Hetzner does is swapping HDDs you identify as faulty (you provide them the S/N of the faulty disk via the support form).

Personally I rent the super cheap Serverbörse servers for personal use. The HDDs in there are fairly old (as it's their older generation servers). Over the past 6 years I needed Hetzner to swap about 5 HDDs. When my server logged issues when accessing a disk, I requested a swap, they swapped it less than 30 minutes later and I started the rebuild of my RAID. All without issues.

> The first one was totally Hetzner's fault since they have failed to identify the faulty disks when they check the disks for it.

Did you mean to say that they removed an incorrect disk when you requested a swap? Or did you expect them to monitor the hardware for you?

Hey Tim,

I totally agree but working on a remote machine it's not always possible, even through KVMs, so Hetzner had to step in on this one.

There were 2 faulty disks from the same 4-disk Raid but once they have found the first, they assumed that first disk was the issue and didn't test the rest, so that ended up with loosing everything.

Even if they are helping as a courtesy, they should test all the disks and do this right before let the customer continue.

I'm afraid, I still don't understand. We are talking about a rented dedicated server here (not about some managed one), right?

> but once they have found the first

How did they find the HDD? As I said: They don't monitor your server. Did you request replacement of the HDD? If you request replacement of one disk, they swap one disk.

> Even if they are helping as a courtesy, they should test all the disks and do this right, before the customer continue.

How should that work? They remove all the HDDs from the server, put them into a different computer to run a test? That would be a privacy nightmare and add unnecessary downtime. I keep track of my HDD status (smartmontools, mdadm Events, ZFS Event Daemon). I know exactly what HDDs I want replaced. They just perform the actual, physical, replacement for me. And that is absolutely fine.

This happened 4 years ago so I found the emails to refresh my memory. Here's what happened according to the emails I have;

1 - I woke up to an unresponsive server in the morning. Couldn't reach to the server in any way.

2 - Got in touch with the support and they asked me to give them like 50 mins to check the hardware and disks and get back to me with;

Dear Client,

We have found this disk S/N xxx defective. Please confirm the full data loss on it in order to replace it.

right after that they wrote;

Dear Client,

We have found this disk S/N xxx defective. Please confirm the full data loss on it in order to replace it.

I asked if they mean data lost on this single disk or the whole array, they said;

Dear Client,

We cannot tell you how is going to react the raid controller as can happens also data loss if it is defective. For that, we recommend before to take Backups of the data. When you are sure to replace the disk tell us, confirm data loss and we will do it.

after many emails and sharing error messages I've got from the raid controller Hetzner said;

Dear Client,

We see our test reported that 2 drives are faulty, but we have replaced one of this only. Now, we could offer to replace the second faulty one (SN: xxx).

Could we do this anytime?

I don't know what kind of test they did but I believe the report he mentioned is from the very first diagnose they did.

I wouldn't know the details since I didn't argue on how they could oversee this, I tried to find a solution.

Edit : I fixed the order of messages.

Thank you for looking that up. Hardware usually does not fail from one day to the next, so to me it appears like your HDD monitoring was insufficient in the weeks / months leading the full crash.

It should have detected the disks going bad and allow you to request a replacement with 3 of 4 disks being good and then later a replacement of the second bad disk with 3 of 4 being good, no?

> I asked if they mean data lost on this single disk or the whole array, they said;

IMO this is the correct answer from Hetzner. They cannot know whether you run RAID 1 (which would be good), RAID 5 (which would have killed the array), RAID 6 (good) or RAID 10 (might or might not be good). Asking before replacing could possibly allow you to save data in case the disk is not completely dead, but rather has a bunch of faulty sectors only.

The only issue I personally see is the bad English. I am German like Hetzner and even I have issues understanding that grammar.

Reply on news.ycombinator.com