Hacker News Reader

Comment by AnthonBerg | Hacker News Reader

I really can’t understand why SD cards are considered applicable as a main storage device for a general-purpose computer. Reliability is the primary bottleneck in everything, and IO speed second. SD cards are the single worst experience I have had in both speed and reliability in any commodity computer stuff for many years.

View on HN

The reason why rPi uses micro SD is that it's the cheapest possible storage solution. You are not buying an industrial-grade workstation for $35. (For those using the rPi in industrial settings, there is the Compute Module which has eMMC. It adds to the cost, though.)

I am not really a fan of "embedded" devices that have to be shut down properly. You can never assume that you won't lose power, and I think there are a lot of flaky/bricked rPis as a result of SD not handing power loss well. But it's cheap, and you just reimage the thing and go when your SD card gets corrupted.

When I last worked on embedded Linux, our OS image was read-only and the writable storage was literally initialized at boot with "mount /dev/mmc1 /storage || mkfs.ext4 /dev/mmc1". That is how you get a consistent state after power loss. But throwing everything away at boot doesn't fit the Raspberry Pi model of being half-embedded half-workstation. The result is flakiness.

Thank you for bringing my attention to the rPi Compute Module, I hadn't known this existed.

> SD not handing power loss well

Is it that SD cards themselves don't "handle power loss well", or is it bad assumptions made by the filesystems chosen to be used on them? Could a filesystem with a specific design avoid SD card corruption?

Partial out-of-order writes can happen with an SD card. Most filesystems do not handle partial writes well, particularly when they happen out-of-order. A redundant checksumming filesystem like ZFS might be better, but that's only because ZFS can totally handle up to N drives per vdev going totally batshit (where N is configurable when setting up the vdev), not because it is designed for SD cards.

Spinning drives can also have misdirected writes, but enterprise drives have for decades now (older than SATA) supported commands for flushing their caches, so proper barriers can be maintained.

My limited understanding from reading articles on the topic is that the SD card is doing things like wear leveling in a way that the OS can't control, so if it loses power while it's doing that, your data is gone. It's like a separate computer, and the integrity of your data requires running the program to completion.

It seems like a bad model if you ask me, but performance is what reviewers publish, not reliability under failing power, so this is what we get. It is obviously possible to fully journal these internal operations and recover them when power is next available (resulting in FS-level errors, instead of internal errors). But the cards do not do this.

With eMMC, you just have the raw flash cells available directly, so you can write this code. With SD, you don't have enough control to write correct code, so you are stuck with what the vendor gives you.

Why does this same problem never occur with USB flash drives? Is it just that a USB device is usually large enough to have room for some power capacitors?

USB is also designed so that data disconnects before power.

I'm not sure if that's enough by itself.

Not if the device loses power, right? I'm guessing USB flash drives will have the same internal mechanisms as SD cards and thus suffer from the same problems.

Oh, are you talking about flash drives used as system drives on a Pi? Because in desktop use power rarely goes out and even more rarely has anything been written recently. In the Pi use case I have no idea.

I have had 2 USB flash drives fail, one definitely after being unexpectedly disconnected (about the other one I don't know).

eMMC is not at all better then SD card though. You really need a real SSD for industrial applications.

It really depends, I think. If you are accessing the eMMC at the lowest hardware level, then you can write OS/filesystem-level code to maintain consistency when power is lost. Meanwhile, an SSD, like the SD card, is doing a bunch of internal operations that the OS can't control, so it's up to the hardware to maintain data integrity. I imagine that SSDs include a capacitor large enough to finish housekeeping, or use integrity-preserving operations internally (journal, etc.), but micro SD cards don't.

There's a lot more to preserving eMMC than just handling power losses.

A lot of what proper SSDs is to do the right wear leveling to make sure that damaged flash blocks don't cause corruption. eMMC is a pretty broad term that can cover something that's just a soldered SD card to something with a full-blown wear leveling controller.

That's why you still see a lot of eMMC failures in embedded systems when someone forgets to not log to the persistent storage or similar wear heavy operations.

> I am not really a fan of "embedded" devices that have to be shut down properly.

the problem is the Pi is not an embedded device, per se.

It's running a desktop operating system that has to be shut down properly. This is the main reason SD cards get corrupted - there is an unchecked flow of writes to the SD card. This increases the chance of outstanding unsynchronized writes and SD card burnout.

I've said many times that raspi-config should allow an option for mounting the filesystems read-only, with an overlay ram based filesystem.

You can run openwrt on the pi, and the filesystem is set up this way. I run a pi like this and have never seen a corruption problem.

>>I've said many times that raspi-config should allow an option for mounting the filesystems read-only, with an overlay ram based filesystem.

I've had success with this:

http://wiki.psuter.ch/doku.php?id=solve_raspbian_sd_card_cor...

yes this was my problem, as well as they actual SD card itself and the SD card slot going out sometimes from frequent rewriting

Would a battery acting as UPS help then? One used just to power the system long enough to properly shut down in case of main power failure?

You don't even need a battery; even a 220uF capacitor gives you a couple of milliseconds at USB power levels[0].

0: USB is supposed to draw < 0.5A @ 5VDC, a resistance of >10Ω. 1F*1Ω=1s, so at 10Ω, each 100uF gets you one millisecond. You do need to worry about voltage dropping off over time though.

That's why I recommended oDroid to people who want to regularly use systems like rPi's. Most of their offerings support eMMC add-ons

If you were to describe and prioritize issues I have with SD cards, "not handling power loss well" would not make my list. Silently dropping writes while fully powered is more like it.

Thanks for the tip. I was wondering about an easy way to boot like this. Better than nothing.

Speaking from SSD firmware development experience, the reason is those devices are so hardware constrained (in particular SRAM and code) that the firmware team spends a lot of time being clever in making it all work under huge time pressure. Couple that with SD cards being a testbed for the newest flash technology. That is why I don't use SD card for anything I care about reliability.

Any alternative suggestions for air-gapped home PC backup?

My primary concern is that those small form factor storage media might not be so reliable for backup. I'd be okay with mSATA , SATA or NVMe form factor from a reputable manufacturer. I always use something one generation behind or been out for atleast an year. Let others test the new NAND and firmware.

How do you airgap a backup?

One way is to have a pool of two or three removable flash devices, with only one plugged in at a time, the others in safe places elsewhere. Automate nightly date-stamped copy to the card of just your essential data. At some interval that strikes a balance between convenience and safety, cycle the cards. You'll also need a scheme for pruning older backups to recover storage.

In that case, understand the SSDs these days don't have great retention specs when not powered. Its one of the drawbacks of using smaller transistors and multi-level cell technology to increase capacity. When powered up they can do a background scan to scrub the data though.

> understand the SSDs these days don't have great retention specs when not powered.

The standard for consumer-grade SSDs is retention of at least a year after the drive is worn out. When the drive has been used only lightly (which is likely to be the case for a backup drive), retention will generally be several times longer. Flash memory is still not a good choice as an archival medium, but it's not like your bits are going to fade away as quickly as a rechargeable battery.

Boot from a read only device, either DVD-ROM. Copy data through a USB data diode.

That is airgapped, but really expensive.

Get a USB sata drive caddy and mount before backing up, unmount afterwards.

Swap the drive.

Is it possible to have a unidirectional USB connection? USB is strictly master-slave.

Yes, just like how it is possible to firewall a TCP connection.

https://www.cru-inc.com/products/wiebetech/usb-datadiode/

no, OP meant filtering write commands

"USB DataDiode" is a brand name of write blocker, all it does is block write(6,10,12) SCSI-commands

https://www.os3.nl/_media/2015-2016/courses/ccf/ccf_tom_fran...

So what is it that makes SD cards unreliable for use-cases such as this?

Is it just the limited amount of flash available for remapping? If so, can one use the same trick as with regular SSDs and simply avoid partitioning some space to improve reliability?

Not OP, but in my experience it's the broad range of SD card quality on the market, and the sketchy/'temporary' physical connection between the gold contacts on the card and the gold contacts in the card reader. In this case, the reliability issues result in the storage device dropping out sporadically entirely during reads and/or writes, or dramatically varying speeds sometimes as low as several KB/s.

Size, Cost, Cross Platform Support and Availability.

Those are very appealing features for sure. My point is specifically that SD cards aren’t actually admissible as a solution - specifically in the context as a main storage device as a general-purpose computer - they are not admissible based only on a cost-benefit ratio because they are broken. Like a glass that is big, cheap, holds any liquid, and is right next to you isn’t admissible if it has a hole in the bottom.

There’s another shadow factor. Probably that they are perceived as the only available thing?

Probably also it is stupid easy for a hardware designer / implementer to tack on an SD card interface.

uSD cards are not broken. You just have to not buy from unknown manufacturers that just put some third party parts together and call it done. And if you use quality power supply and wiring, to avoid brownouts, SD cards can be quite reliable. Choosing an appropriate filesystem may also help (like f2fs).

I have several multigb PostgreSQL databases (with checksumming enabled) running from uSD cards for years, scraping data from the web hourly.

I find uSD cards quite reliable, actually (for rootfs use), in contrast to what's being said online all the time.

I also like that it's removable. If it breaks you just pop in another $4 32GB card, restore, and continue where you left off. Compared to soldered on eMMC, that's much more user friendly.

I did all the correct things and encountered all the failures described online :)

It’s heartening to hear that they’re really, actually workable.

In a 5 - 55 dollar device, they make a lot of sense. If your application is such that SD isn't good enough, the rpi itself probably isn't either. Both are great for the price to capability ratio. Of course they won't last forever, but you don't complain about not getting ceramic mugs at the local takeout coffee joint, either.

I wish rPI could boot from network

The RPi 3 and late revisions of the RPi 2 support PXE on the built-in Ethernet adapter: https://www.raspberrypi.org/documentation/hardware/raspberry...

The RPi 4 has a boot process with a programmable boot ROM that should allow for a more "real" bootloader like uboot in the long term, but currently PXE isn't supported.

Reply on news.ycombinator.com