Hacker News Reader

Comment by 01100011 | Hacker News Reader

For me, the increased perf is nice, but what I really want is increased longevity. IME, microSD is a poor fit for hosting a R/W filesystem. Every time my Pi based systems have gone down, it's due to a faulty microSD. I was using well rated microSD cards and they still died relatively quickly(months, not years).

I suspect they'd be fine if I was running a read-only rootfs, but with a lot of the applications I run(i.e. HomeAssistant) you need R/W capability.

View on HN

log2ram [1] can help you with reducing the writes of logging at least. It saves logging to ram and then writes it once a day.

[1] https://github.com/azlux/log2ram

This is a beautiful little utility.

I've been doing stuff like this manually for years, and usually throwing away what gets logged.

Thank you for sharing this, it's comments like this that make it hard to give up skimming HN. I have two Pi's running on SD cards that this is immediately beneficial for, so thank you.

/tmp and /var/log on tmpfs works as well, though it needs a bit of fiddling.

The microSD cards usually just get damaged when writes are happening (and this increases with power fluctuation e.g. when you are powering a witless-adapter, zigbee dongle with the Pi). Very likely if you run something like HomeAssistant.

The best way to avoid this (without having a pci-e ssd connected to the Pi) is booting from a read-only FS on the SD card and then mounting a USB connected SSD for writing data.

If you want to have a "fake" R/W system take a look at https://yagrebu.net/unix/rpi-overlay.md which is a R-only FS which writes changes into RAM only. I run my PIs with that.

I made some flash-based appliances many years ago. There are some things you can do to reduce the number of writes. Have you tried mounting your root fs with `noatime`? Otherwise reads will generate a write to your device (to update the access time).

You can boot off of usb and use a better (and faster) flash drive.

USB commonly has signal integrity issues. Barring those, the connector can just get jostled and poof there goes storage.

> USB commonly has signal integrity issues.

That's simply not true.

Yes, it is. Everything about the USB 2.0 PHY is out of control for a 480 Mbps line rate. Things were tightened up in USB 3.0 just as they were in the move to USB 2.0, but not nearly enough to guarantee low BER in real world scenarios at advertised rates. USB is made to be cheap, not reliable.

I do multi-terabyte backups to USB3-connected devices and networked devices. I'm in the habit of using a merge tool for the backups, and using it again to verify that the backups are correct. One thing I've found is that the incidence of flipped bits is extremely small relative to the amount of data transferred, but real, with both USB3 storage devices and network shares accessed via ethernet.

Do they use the same error correction algorithm, perhaps? (I honestly haven't looked into it much) An alternate possibility is that the USB3 and ethernet transport error correction is working fine and it's the devices that are not storing data accurately... that seems less likely to me, but somewhere in there is a SATA bus that also has to do error correction.

In any case I don't feel great about a statement like "USB commonly has signal integrity issues" since the way I work wouldn't be possible if it were true. It's not perfect, for sure, but I honestly feel very good about my USB-connected SSD storage relative to so many other things I've used in the past: various optical drives, floppies, SD cards, and some not-great hard drives.

USB doesn’t have error correction. It has a simple CRC and retransmit behavior, which buys a few orders of magnitude on the BER, but is a far-cry from something like FEC. Typically you would want a higher layer to do error detection and correction for USB devices.

The quality of the PHYs and media matter greatly. If you’re not using the cheapest PHYs, aren’t in a noisy environment (ie many USB devices connected to the same controller), and have high quality cables/connectors (that are also short and far away from crosstalk aggressors), then you may never see errors.

Something working well in a cozy scenario doesn’t make it a good solution. I could run an out of spec 2 meter cheap SATA cable in a bundle and never see errors. What matters is BER margins at real world scenarios. USB should be implementing more aggressive error correction at layer 1 for what they’re doing. It shouldn’t be the job of higher layers to handle signal integrity issues.

I think on these points I agree. Given how fast and efficient even the cheaper microcontrollers have become there's not a reason for any new spec to avoid the highest quality error detection and correction.

It's fair to point out that my decent results with this equipment might partially be down to a pretty RF-quiet environment, avoiding USB hubs for all the reasons a person might want to avoid USB hubs, avoiding piling on devices, and so on. Trying to make some of this equipment fail would be an interesting thing to experiment with someday.

in the context of the pi, USB > MicroSD in all the metrics that matter

Not a pi user. Why not SATA? Is it not supported on pi yet?

Pi's don't have a SATA controller or SATA ports. NVMe doesn't need a controller; it runs over a PCIe lane.

Why on earth would you use the microSD when you were booting off an nvme drive, as is the whole point of the OP?

I think they're saying it's the longevity, rather than the performance boost, that they're excited about.

That's how I read the grandfather comment too—a 3-4 GB/sec NVMe SSD wastes a lot of performance potential on a Pi, but the main thing that is nicer about even low-end SSDs is they have way better write/rewrite longevity than microSD cards, few of which are design in any way for general computing use.

Most microSD cards optimize for only one thing—storing large, sequentially-written video files, then copying them off. Rinse and repeat.

Even some of the fancy "100 MB/sec+" microSD cards only get like 1-2 MB/sec in random access patterns.

4k block random read on a regular A1 rated uSD card is 8MiB/s. I benchmark 11MiB/s on some of my Sandisks Ultras, depending on manufacturing date. Writes usually don't matter much, unless you use an inappropriate filesystem for a SD card or sync a lot.

Longevity is longer for SSDs but it's not that bad for uSD cards either. If you setup your system so that there are no pointless writes all the time, they last 3-5 years, no problem, and probably more.

The biggest benefit is that uSD cards are much lower power. And unless the use case is very storage heavy, like a constantly accessed database, or backups, SSD is just a waste of resources.

> If you setup your system so that there are no pointless writes all the time, they last 3-5 years, no problem, and probably more.

Define pointless writes.

Anything not related to your use case for the board. Random crap general GNU/Linux distros will do because this is not an issue, on usual HW.

Consumer mircoSD cards commonly have firmware bugs. A Raspberry Pi running Linux will consistently trigger these bugs given enough time, even if you mount the card read-only. The nand is fine, but the card will be unreadable until reformatted. You might never see this from a single Pi running at home, but people who deploy large fleets of Pis inevitably run into these problems. Industrial microSD cards aren't perfect, but they are more reliable.

Removing an SD card from its socket during a write operation can actually brick the card enough to no longer work at the block level. Perhaps it's brand dependent, but I'm talking about a well known name brand. It's rare, but I know of a ziplock bag of bricked SD cards that shows it's not too rare.

I suspect it occurs when a card loses power while in the middle of shifting data around for wear leveling purposes. Presumably, some internal data structure gets borked in a way that the card's embedded MCU firmware didn't anticipate and can't recover from.

It doesn't even have to be during a write operation. Nand needs to be periodically refreshed after a read, so the controller can be shuffling data even on a read-only mounted card. I've read speculation that most consumer cards store their controller firmware on the same nand that is used as the general storage, and thus are prone to corrupting their own firmware if something goes wrong while shuffling data.

Power loss is definitely a threat to an SD card in a Pi. But I've seen many cards go bad while the system is still running, so it's not the whole story.

Weirdly, I've had a number of cards that entered a state where they would be inaccessible under Linux or Mac OS. But they could easily be recovered by a re-format under Windows. No idea why. Perhaps the manufacturers test a lot more thoroughly with Windows?

That's a good hypothesis re: the MCU firmware on the NAND. If it's true that all NAND needs to be periodically refreshed, then it could be as simple as losing power while one of the firmware sectors is getting refreshed. That seems like an obscure enough scenario that there's dragons lurking around it.

I like to imagine in my head that the MCU in an SD card can detect its voltage dropping quickly enough to get into a safe state most of the time. With all the recommended capacitance around the socket, it might actually have a few milliseconds to work with after external power loss. When you pull the card out of a socket, though, the MCU stops working before it has time to do anything about it.

That's probably true. I don't run one board though. I run ~12 24/7 for years. I also run many other devices from uSD cards that get turned on/off hundreds of times a day on occasion, when I'm doing some development, including forced power cuts.

I didn't have many issues with my consumer cards of choice, which are Sandisk Ultra A1, which I chose because they are cheap, not too slow and not too fast (interface limit is 23MiB/s and there's no point using faster cards), made by the actual manufacturer of the NAND chips not just some packager buying random NAND chips from a bucket on the spot market, and have verifiable origin via scratch codes.

Not sure about their failure modes, because all of the cards I had fail on me so far were Samsung ones. (some/most of them probably fakes, because I bought them on aliexpress, and Samsung didn't give a crap about consumers being able to verify origin of the card, until very recently and for select cards only)

So far I'm very happy. Cheap and reasonably reliable low power storage.

I guess if I had 10000's of cards in the wild in harsher conditions, I'd care if 1/100 failed on me per year. But in home conditions, 1/100 per year means 1 card failing every 2-3 years in any of my devices, which is very tolerable, and easy to recover from.

Reply on news.ycombinator.com