Hacker News Reader

Comment by ars | Hacker News Reader

Note: This article is from 2007 and is quite prescient.

It's completely shameful how bad the specified read error rates are now.

It's to the point that if you read an entire disk 4TB you have a 32% chance of one bit being wrong!

That means hard disks can no longer be considered reliable devices that return the data written to them, you now need a second layer in software checking checksums.

http://www.wolframalpha.com/input/?i=4TB+%2F+10^14+bit++in+%...

For extra money they sell hard disks with 10^15 reliability instead of 10^14 - this should be standard!

View on HN

I think that it's more shameful that modern operating systems are only slowly adapting file systems that can even detect that silent data corruption has occurred. EXT4, HFS+, and NTFS all make no effort to ensure the data you read back is the data you wrote.

Slowly Btrfs on Linux and ReFS on Windows are becoming more prevalent. Since storage density is growing faster than most home users can use it, hopefully in the near future consumer computers will come configured to store multiple copies of their data. Instead of one day noticing that a picture is corrupted or the computer won't boot, periodicity scrubs will detect corruption before it's too late.

> It's to the point that if you read an entire disk 4TB you have a 32% chance of one bit being wrong!

If this were the case, forums would be filled to the brim with ZFS users whinging about how they get CKSUM errors practically every time they scrubbed their pools. They do happen, but nothing like that frequently.

> That means hard disks can no longer be considered reliable devices that return the data written to them

They were never such devices in the first place if you ever valued your data enough to care. Even if your disks are perfect, your cabling, backplanes, IO controllers, drivers or memory probably aren't.

> If this were the case, forums would be filled to the brim with ZFS users whinging about how they get CKSUM errors practically every time they scrubbed their pools. They do happen, but nothing like that frequently.

I'm going by what the manufacturers say - are you saying they are overestimating the error rate? Why would they do that?

> They were never such devices in the first place

No, I meant they would return false data instead of an error. Used to be you never got corruption, just success or fail. Now you have success, false success, or fail.

> I'm going by what the manufacturers say

Are you really? "Non-recoverable Read Errors per Bits Read, Max", say Seagate - that just means it couldn't correct, not that it couldn't detect there was an error.

> are you saying they are overestimating the error rate?

Perhaps not, as a general order-of-magnitude upper-bound on a marginal drive in poor conditions, but certainly if you're interpreting it as an average. Too many people would be seeing it happen way too often to keep quiet about it otherwise.

> Used to be you never got corruption, just success or fail

I wish :/

I think we have very different definitions of 'completely shameful'. One bit wrong in four trillion, on a device that only costs a couple of hundred dollars?

It sounds good doesn't it? But then you compare to the size of the disk and realize that it's not actually that good.

The shameful part is that they actually sell the 10^15 drives - and not for a ton more money either. They simply bin drives and the better ones are rated as 10^15. Instead of doing that they should figure out what's different in the manufacturing of those and do it across the board.

Yeah... shameful. I am sure that there is something simple that all the drive manufacturers are just overlooking in their process that would take them to 10^-17 BER.

When was the last time you created a physical artifact for sale? That had even just part per billion malfunction rates?

It isn't as easy as these guys make it look by a long stretch.

Even worse/better -- one bit wrong, not byte. So one bit wrong in thirty-two trillion.

lol @ "they should sell hard disks with 10^15 reliability instead of 10^14!" This is the funniest outrage I've seen in a while, just because the suggested fix is so specific and (relatively) low for the huge amount discussed.

It's like saying, "It's an outrage that this drink holds only 37 ounces! For this kind of money I expect it to hold at least 38 ounces. I would pay for that."

My 4 downvoters are clearly a lot worse at math at I am.

Let me spell it out for you:

"Reliability of 100000000000000 is unacceptable! I need exactly 1000000000000000 of reliability (a factor 10 difference only)."

If you don't see how that is funny, the problem is with you, not me. That's a hilariously specific, and small, change.

"Man, having a 30% chance of a bit flipping if I read 4 TB is unacceptable. I should have to read like...40 TB for a 30% chance of a bit flip!"

It's particularly funny because in the face of this being unacceptable, I would expect a solution, like error correction, that adds 5-10 orders of magnitude (many bit flips have to all happen in the same sector, by coincidence, at the already low 10^14 rate), for it to make it past the error correction layer.

The other thing is that given that we're talking about ERROR RATES, I cannot possibly believe that they have it down to such a specific number as 100000000000000. It would have to be a range. You should target more than 1 order of magnitude to get something more reliable (like ECC RAM versus regular RAM). [1]

Just like the error margin on 37 ounces must be like 37-40 ounces anyway.

[1] first reference I found: http://lambda-diode.com/opinion/ecc-memory-2

"I computed that if you have 4 GiB of memory, you have 96% chance of getting a bit flip in three days because of cosmic rays. SECDED ECC would reduce that to a negligible one chance in six billion." That is, for example, a change of

1 per 34359738368 bits (rounding 96% to 100%) to 1 per 206158430208000000000 bits (multiplying by 6 billion).

34359738368 bits is 10^10.

206158430208000000000 is 10^20.

That's the kind of change (give or take) that I was expecting parent to suggest - not one order of magnitude!

Clearly math isn't your strong suit.

An order of magnitude change is nothing minor. It would take the 32% chance and make it a 3.2% chance.

Maybe you don't know that ^ means to the power off?

Hey, please remember to be kind to everyone. Even if you disagree with someone, insulting them adds no value for them and adds no value for the other readers on HackerNews. Insults are the rare type of comment that add value for only a single user. Even if someone is wrong, remember: we're all human. :)

I know, you're right. I was just a bit annoyed at his tone so I reacted instead of just replied.

I tried to soften it a bit by adding if perhaps he just didn't recognize the notation, but I guess it wasn't enough.

I did recognize it, ars. I think it's hilarious that if 100000000000000 (1e14) isn't enough reliability then 1000000000000000 (1e15) should do it.

As mentioned if you look at, for example, the change from normal bit flips (from cosmic rays) on normal RAM, versus that on ECC RAM, you would see many orders of magnitude. Given that we're speaking about errors (which have a huge standard deviatoin anyway) it is quite normal to see orders of magnitude difference. A factor of 10x is just not that big of a wish, if the error levels are causing you problems!

I also don't think that multiplying by the size of the drive is relevant to anything. It doesn't increase the chance you have an error in a specific application, etc.

What can you fill 4 TB with, that you wouldn't use software error correction on, but for which a single bit flip is unacceptable?

I can think of things like movies, raw sensor data, and other media. I don't see how multiplying by the size of the drive gives you an interesting result. You've just fallen for marketing :)

> should do it.

They actually sell 10^15 disks, so there is no excuse for 10^14 ones to exist anymore. They aren't even using more error correction to get it - they are binning drives and the better ones get sold for more.

This would be fine is the lower spec'd ones were still reasonable, but for a 4TB drive 10^14 is NOT OK, and they should not be selling them anymore.

I just disagree with you and think that your perspective on the matter and use of the numbers is funny. Again you repeat "For a 4TB drive 10^14 is NOT OK"... but 10^15 is. Well, if you say so.

Note: you shouldn't downvote just because you disagree with someone's perspective. I'm not going to delete these comments because I stand by them. If 1 bit flip every 4 TB is a problem for you, then don't move to 1 bit flip every 40 TB - get some kind of ECC etc.

He's not saying that 10^15 is good enough for all purposes. He's saying that it's closer to good enough without significantly increasing the cost. This should not be that hard for you to grasp.

Your tone with me is completely unwarranted, wtallis. This is what he wrote:

Note: This article is from 2007 and is quite prescient.

It's completely shameful how bad the specified read error rates are now.

It's to the point that if you read an entire disk 4TB you have a 32% chance of one bit being wrong!

That means hard disks can no longer be considered reliable devices that return the data written to them, you now need a second layer in software checking checksums.

http://www.wolframalpha.com/input/?i=4TB+%2F+10^14+bit++in+%...

For extra money they sell hard disks with 10^15 reliability instead of 10^14 - this should be standard!

Even after his (and your) clarification, it makes me chuckle to read that :) He started, "It's completely shaemful how bad the specified read error rates are now" and then ended "For extra money they sell hard disks with 10^15 reliability instead of 10^14 - this should be standard!"

This is hilarious. I guess some people here have no sense of humor though.

Error rates are not nearly as exact as that - more like marketing figures.

Does this mean that a 10^13 reliability has 320% chance of failure?

EDIT: in any case the correct calculation should be http://www.wolframalpha.com/input/?i=%281-%2810^-14%29%29^%2... which gives a 30% chance of failure, versus http://www.wolframalpha.com/input/?i=%281-%2810^-15%29%29^%2... for a 3.5% chance.

320% would mean you find 3 wrong bits each time you read the disk.

I was calculating the expected number of wrong bits per full read of the drive.

You calculated the chance of a perfect read which is similar but not the same.

Ah, I see. Could you edit your post to reflect this? As it stands it is very easy to misinterpret.

You can only edit for a little over an hour, and it's been three.

I thought "of one bit being wrong" said it, but I can see how you misread it. Hopefully people will read the thread and see these replies as well.

But only a few years of difference with Moore's Law... Which was the point of the article.

You probably mean: https://en.wikipedia.org/wiki/Kryder's_Law

I have now spelled it out for you. "Reliability of 100000000000000 is unacceptable! I need exactly 1000000000000000 of reliability (a factor 10 differnece only)." It's hilariously specific. If your error rate is riding up against your usage so that you notice it, you wouldn't expect to just want to cut it by a small factor like that.

> It's hilariously specific.

The reason for the number is very simple: It actually exists.

I think it's probably just marketing. They're just engineering reliability targets, a specific batch could have far different rates.

And clearly it isn't your's either.

I had the same thought logicallee :)

Reply on news.ycombinator.com