Hacker News Reader

Comment by eek2121 | Hacker News Reader

I see a ton of replies here. I am a tech user with thousands of terabytes stored locally. Most of those terabytes are mirrored in the cloud. In all of those situations I've never actually had any corruption related to memory or storage. I have 932gb in PICTURES ffs.

I get the claimed need for EEC, but yet I don't. I own a number of web properties and regularly update logos, software packages, html files, PHP files, ruby files, etc. No corruption.

View on HN

How do you confirm that there's no corruption? My understanding was a silent bit flip would be difficult to detect. I ended up using ZFS and ECC memory to avoid silent corruption, but I'm largely relying on the motherboard to do the right thing when an ECC error is detected. It'd be nice to verify after the fact.

git fsck

On 932 GB of binary data? I'm probably operating on old impressions of git, but that sounds like it'd be a nightmare to work with. Ideally, I'd like something that runs in a reasonable amount of time so it can be run on a regular schedule. I currently run a ZFS scrub that takes about 90 hours to finish on 32 TB of data, so it's only run monthly.

bit flip hard to detect - true for RAW, false for .jpg or other compressed files.

If your pictures are not RAW but compressed (say jpgs) then the corrupted photos will really jump out at you - half the picture is there, but then noise. I used to make sure I was storing jpgs not RAW, but then learned that wasn't always a good thing, several times over. Dunno whether damage happened in RAM or on HD, but I strongly suspect RAM.

The bad habit that nailed me was probably this one: I would ball up a whole bunch of photos in a big .zip; (to enhance copying speed and maybe privacy) then next year unzip that, add that year's selected photos and zip it again. Year after year. That added up to a lot of trips through RAM in which one bit flip could take out an image. Bugs in compression software aren't impossible as the villan, but that's not my bet.

Solution: I now zip up a few years together and copy each chunk I don't ever rezip old large chunks. No new errors noticed.

You can be pretty sure that you have some corrupt bits

No, we really can't be sure. I'm annoyed when people are sure about it without posting their ECC correction rate stats from live systems.

I'm continuously writing metrics on a non-ECC system. There are 3 places that the bitflip can affect the data: pre-writing (checksum will be correct), when flushing to the drive (checksum mismatch), calculating the checksum (mismatch again). There may be some silent corruption (in 1 out of 3 cases), but I've been scrubbing the drive every day for ages and there's not a single error.

Non-ECC seems to be doing well. Other claims could really use actual numbers.

Reply on news.ycombinator.com