Comment by mrsteveman1 | Hacker News Reader

Comment by mrsteveman1 | Hacker News Reader

Parent Full thread

A small amount of it, yes, but it's the card itself causing corruption issues most of the time, it happens even on devices where the filesystem is read-only. The controller will write and move things around for maintenance purposes even if the host hasn't issued a write command.

The cheap SD cards just aren't designed for anything except being used in consumer devices with batteries, where sudden power loss is rare and losing data isn't going to cause a plane to crash or result in someone not receiving a dose of insulin.

So when they suddenly lose power, they aren't always capable of ensuring that whatever task they were carrying out at the time is actually completed and did not accidentally destroy data.

And apparently the consumer SD card controllers are really there to manage and remap parts of the flash that were defective before ever leaving the factory.

It's probably cheaper to build over-provisioned cards with a simple controller that can deal with manufacturing defects in the field, than to do QA on 200 million thumbnail sized NAND die every month and still try to profit while selling them for a fraction of a penny each.

It seems very much possible to fix data corruption of a broken flash translation layer of SD cards with yet another translation layer using erasure coding. Whatever the controller does or doesn't it still can only write in blocks, we just have to make sure erasure codes are at least a block size apart from the data.

At least for me, the failure mode in most cases had the card simply erroring out on accesses (even writes!) to particular sectors. Other sectors would allow to write, so it wasn't that the card had run out of sectors with write lifetimes. When you're not given anything to work with, even corrupted data, extra ECC isn't going to do you any good. My conjecture (with no data) is that the card's FTL was itself corrupted.

EDIT: and in no cases I saw did the cards I was testing give me truly 'corrupt' data. Just either error codes, or stale data, or occasionally data from another sector entirely. They've got metric shittonnes of ECC internally (to make up for the crappy NANDs), and will do a better job than you can at detecting errors.

Right, so, if we use virtual blocks the size of the physical blocks and either parity block or one of the data blocks are destroyed it is still possible to get the data. And with writes if a write fails our layer can just try to write a bit farther and farther until it succeeds.

In general you don't have access to the physical block size. And it can change even within a given lot.

Exact block size is not necessary, big safe choice is fine too. But I suspect it's trivial to detect it with benchmarks, as writing even a byte over a block size would require overwriting two physical blocks instead of one, which is much slower.

Reply on news.ycombinator.com