Hacker News Reader

Comment by lomnakkus | Hacker News Reader

Indeed. It's true that the data may be corrupted before hitting any disk[1], but once it has hit the disks (>1), it's extremely unlikely that you'll ever hit a similar bit error where it'll mistakenly choose the wrong disk block to recover from.

The main point of e.g. ZFS or Btrfs checksumming is that a) at least it isn't getting worse, and b) I can tell if it's getting worse.

[1] ... but if the bits are not generated by the machine that actually saving to disk, how do you know they weren't corrupted along the way? The number of people who religiously check PGP signatures/SHA256sums or whatever is miniscule.

View on HN

> The number of people who religiously check PGP signatures/SHA256sums or whatever is miniscule.

• If you transfer things around using BitTorrent, it'll ensure you always end up with a file that hashes correctly to the sum it originally had when the .torrent file was constructed.

• Many archive formats (zip, rar, and 7z, at least) contain checksums, and archival utilities validate those checksums during extraction, refusing to extract broken files. "Self-extracting archive" executables that use these formats inherit this property.

• Some common disk-image formats (dmg, wim) embed a checksum that checks the whole disk-image during mount, and will refuse to mount a bad one. (I believe you can then try to "repair" the disk image with your OS's disk-repair utility, if you have no other copies.)

• Web pages increasingly use Sub-Resource Integrity attributes on things like .css and .js files, protecting them (though not the page itself) from errors.

• ISO files don't embed checks, but all the common package formats (Windows .cab and .msi; Linux .deb and .rpm; macOS .pkg) on installer ISOs embed their own checksums and often signatures.

• git repos are 'protected' insofar as you won't be able to sync mis-hashed objects from a remote, so they won't spread.

Really, looking over all that, it's only 1. plain binary executables, and 2. "media files" (images, audio, video)—and only when retrieved over a "dumb" protocol, rather than a pre-baked-manifest protocol like BitTorrent or zsync—that are "risky" and in need of explicit checksum comparison.

True. Still mostly immaterial in the grand scheme of things, I think?

EDIT: Should say, as a point of interest: Even though .zip's were protected back in the Good Old Days, that didn't really matter because we all got corrupted (expanded from .zip) .mp3's because of those fucking RTL3xxsomething cards that would just transmit things perfectly and then corrupt the checksum (or whichever way 'round). Ugh. One of the few times I've actually hated engineers.

(Don't get me wrong. I really do want more of these checks to be pervasive. We start with our local file systems.)

Both macOS/iOS and Windows use code signing for executables, which should guard against most types of corruption.

Web pages (and anything else) transmitted over HTTPS are protected from corruption in transit by TLS's hashing (which is vastly stronger than the checksums at lower levels of the network stack), though that doesn't help if the server has faulty memory or storage.

PNG has built-in checksums, though other image formats don't (JPEG). Not sure about video.

> Web pages (and anything else) transmitted over HTTPS are protected from corruption in transit by TLS's hashing (which is vastly stronger than the checksums at lower levels of the network stack), though that doesn't help if the server has faulty memory or storage.

I didn't bring this one (or any other transport-level checksums) up, because we were talking about whether you can trust something "across the whole process"—from its origin developer's disk (where it might get an initial explicit checksum generated), to origin memory, across the network to a server's memory, to that server's disk, over the network again to a CDN reverse-proxy's memory, maybe its disk, then the network again to you, then your memory, your disk, and finally your memory again as you verify it. Oh, and a bunch of routers and switches in between, of course.

Static checksums that are baked into file formats or manifest files protect the file across that whole chain. Transport-level checksums only ensure that the one part they're involved in happened correctly.

What are you doing where you're actually checking checksums periodically and detcting when things get worse? That seems like a lot of work to set up.

They are using ZFS, scrubbing is one command.

zpool scrub

(This may be a myth: It's not something you should actually do that often because actually reading the media may dregrade it.)

Either way, it'd be fine on SSDs, then?

Hm? As long as the error correction technology on your chosen SSD of choice stands up, I guess... yes? What, exactly, are you asking?

My thought was "reading SSDs doesn't degrade them, so there's no disadvantage to constant scrubbing." Unless I've misunderstood what you mean by "reading."

Ah, right, fair point. AFAIUI SSD storage does degrade a tiny(!) amount when reading, but magnetic storage degrades quite a bit more.

(That's where that came from, draw your own conlusions :).)

or for btrfs users out there:

btrfs scrub start /mnt/volume_name

Reply on news.ycombinator.com