Raspberry Pi microSD follow-up, SD Association fools me twice?
jeffgeerling.com
jeffgeerling.com
I am not really a fan of "embedded" devices that have to be shut down properly. You can never assume that you won't lose power, and I think there are a lot of flaky/bricked rPis as a result of SD not handing power loss well. But it's cheap, and you just reimage the thing and go when your SD card gets corrupted.
When I last worked on embedded Linux, our OS image was read-only and the writable storage was literally initialized at boot with "mount /dev/mmc1 /storage || mkfs.ext4 /dev/mmc1". That is how you get a consistent state after power loss. But throwing everything away at boot doesn't fit the Raspberry Pi model of being half-embedded half-workstation. The result is flakiness.
the problem is the Pi is not an embedded device, per se.
It's running a desktop operating system that has to be shut down properly. This is the main reason SD cards get corrupted - there is an unchecked flow of writes to the SD card. This increases the chance of outstanding unsynchronized writes and SD card burnout.
I've said many times that raspi-config should allow an option for mounting the filesystems read-only, with an overlay ram based filesystem.
You can run openwrt on the pi, and the filesystem is set up this way. I run a pi like this and have never seen a corruption problem.
I've had success with this:
http://wiki.psuter.ch/doku.php?id=solve_raspbian_sd_card_cor...
Is it that SD cards themselves don't "handle power loss well", or is it bad assumptions made by the filesystems chosen to be used on them? Could a filesystem with a specific design avoid SD card corruption?
It seems like a bad model if you ask me, but performance is what reviewers publish, not reliability under failing power, so this is what we get. It is obviously possible to fully journal these internal operations and recover them when power is next available (resulting in FS-level errors, instead of internal errors). But the cards do not do this.
With eMMC, you just have the raw flash cells available directly, so you can write this code. With SD, you don't have enough control to write correct code, so you are stuck with what the vendor gives you.
I'm not sure if that's enough by itself.
Spinning drives can also have misdirected writes, but enterprise drives have for decades now (older than SATA) supported commands for flushing their caches, so proper barriers can be maintained.
0: USB is supposed to draw < 0.5A @ 5VDC, a resistance of >10Ω. 1F*1Ω=1s, so at 10Ω, each 100uF gets you one millisecond. You do need to worry about voltage dropping off over time though.
A lot of what proper SSDs is to do the right wear leveling to make sure that damaged flash blocks don't cause corruption. eMMC is a pretty broad term that can cover something that's just a soldered SD card to something with a full-blown wear leveling controller.
That's why you still see a lot of eMMC failures in embedded systems when someone forgets to not log to the persistent storage or similar wear heavy operations.
That is airgapped, but really expensive.
https://www.os3.nl/_media/2015-2016/courses/ccf/ccf_tom_fran...
Swap the drive.
The standard for consumer-grade SSDs is retention of at least a year after the drive is worn out. When the drive has been used only lightly (which is likely to be the case for a backup drive), retention will generally be several times longer. Flash memory is still not a good choice as an archival medium, but it's not like your bits are going to fade away as quickly as a rechargeable battery.
Is it just the limited amount of flash available for remapping? If so, can one use the same trick as with regular SSDs and simply avoid partitioning some space to improve reliability?
There’s another shadow factor. Probably that they are perceived as the only available thing?
Probably also it is stupid easy for a hardware designer / implementer to tack on an SD card interface.
I have several multigb PostgreSQL databases (with checksumming enabled) running from uSD cards for years, scraping data from the web hourly.
I find uSD cards quite reliable, actually (for rootfs use), in contrast to what's being said online all the time.
I also like that it's removable. If it breaks you just pop in another $4 32GB card, restore, and continue where you left off. Compared to soldered on eMMC, that's much more user friendly.
It’s heartening to hear that they’re really, actually workable.
The RPi 4 has a boot process with a programmable boot ROM that should allow for a more "real" bootloader like uboot in the long term, but currently PXE isn't supported.
This is the disk equivalent of 'buffer bloat'[1] but with disk data. The question you have to ask is what happens when you lose power and there are 'n' writes queued to the storage device. How has the kernel marked that data? Historically once the disk accepted the write and returned kernels would say "ok, data on disk" buffer clean now.
Of course if you lost power and that write never actually completed, well you now had file system corruption (possibly silent if it was just a data block).
Another problem that cropped up is "ok we can re-use this data buffer" so it gets fill with different data, and then the storage device says "oh hey, send me that data you told me write ..." whoops.
The third problem that crops up is that the computer on the storage device crashes (or watchdog resets) and it goes back and doesn't remember what commands it had in the queue. So some time later your kernel better ask "hey why haven't you acknowledged this write we sent 50mS ago?" and have the disk say "What write?" and replay it.
One school of thought was "Command queuing in the storage device is always bad, the kernel knows more, has more memory, and is ultimately responsible for what is and what is not on disk." So the kernel would 'hold' writes to the disk until the write was sufficiently aged or until enough writes had accumulated to do a "streaming" write (multiple sectors on the same track). But for non-spinning media there is no seek time so there is no advantage. Except for flash, you have to erase a page which can be a lot of data which then has to be rewritten with the new stuff.
I enjoyed the writeup but it really needs to start with a highly fragmented SD card (one with lots of random I/Os to it) so that it can measure the latency hit for page rewrites.
[1] https://www.bufferbloat.net/projects/bloat/wiki/Introduction...
Additionally operating systems also play exactly the same "dangerous" game with writes, they keep dirty pages in memory and flush them out gradually. You are only guaranteed that things actually got to disk when you fsync them which sets off a cascade of writes and flushes if they didn't already happen in the background.
Write performance is largely a very well-maintained illusion if you don't have something like Optane. If you open files in SYNC mode write performance craters, even on NVMe.
What filesystems do to maintain sanity among this madness is to order writes so that if a crash occurs you lose a few seconds of data, but you don't lose consistency, you just get an older state of whatever made it to disk.
SD cards are not special in this regard.
Is there a reason why storage devices can't accept the write, and then notify the kernel when the the write is complete? I'm fairly certain that's how it works right now.
>The third problem that crops up is that the computer on the storage device crashes (or watchdog resets) and it goes back and doesn't remember what commands it had in the queue. So some time later your kernel better ask "hey why haven't you acknowledged this write we sent 50mS ago?" and have the disk say "What write?" and replay it.
Don't you run into the same issue if you were doing writes and the user yanked out the power? What's written after the last fsync is pretty much undefined behavior and it's up to the application/filesystem's journaling mechanism to deal with it.
>One school of thought was "Command queuing in the storage device is always bad, the kernel knows more, has more memory, and is ultimately responsible for what is and what is not on disk." So the kernel would 'hold' writes to the disk until the write was sufficiently aged or until enough writes had accumulated to do a "streaming" write (multiple sectors on the same track). But for non-spinning media there is no seek time so there is no advantage. Except for flash, you have to erase a page which can be a lot of data which then has to be rewritten with the new stuff.
So command queuing is bad except on hard disks and flash. Then what else is left? Those two cover 99.99% of consumer storage.
To head off replies about fixed time garbage collection, sure it exists, but you cant guarantee it will reclaim enough space in a fixed timeslot (think of a massively pathological case where 99% of every flash chip is full, but super fragmented and data needs to be shuffled around perhaps hundreds or even thousands of times before enough space can be reclaimed)
That's why you have spare space. It's not overly hard to clamp write amplification to a number like 10x or 20x, even in the absolute worst case.
Note that the SD standard here requires a minimum number of writes per second inside a 256MB area, and even 1% spare space on a 32GB card would give you a >256MB buffer.
I wish the pi just had a 12v barrel jack.
edit: it first dawned on me that the pi power situation could be improved when I read this article years ago:
http://www.bitwizard.nl/wiki/Reducing_power_consumption_of_a...
While it's not quite as beefy as the Pi (especially the Pi 4), nor as cheap, the Beaglebone family of boards are awesome. They come with schematics, and you can order the processor in qty 1 from Digikey. I haven't yet remixed one yet (a 400-BGA is going to be stretching my abilities...), but I have leaned on the schematic quite a bit to figure out some clever things I can do with it.
One thing he pointed out to me was the pin design. Everybody else would design it with female pins so that you won't accidentally bend the pins or short them out.
He loved the beaglebone though and recommended it.
That said, I think the pi has a bigger community and more software. (I might be wrong)
Most definitely. From an "ease of getting started" perspective, the Pi is absolutely amazing. NOOBS, or whatever the distribution is called, is pretty painless to get going. You flash it to the SD card, plug in a monitor and keyboard+mouse, and you boot straight into a familiar GUI with tons of tools pre-installed. Plus, the $35 price point (well, more once you include SD card, power supply, etc) is pretty hard to argue with.
Debian IoT (the "standard" distro for the BeagleBone) is cool in its own right, but definitely not as approachable for someone who's new to embedded Linux. Things like GPIO mappings need to get tweaked sometimes, you're dinking around with device trees here and there, etc. For the stuff I've been working on lately (robotics), the BeagleBone Blue has been an absolute godsend. Tons of ports that speak different protocols (UART, I2C, PPM for driving servos, GPIOs, etc), and librobotcontrol is really straightforward to start building with.
I guess a great way to distinguish between them: I pretty much always use a keyboard and mouse to get a Pi set up. I don't even know if the BBBlue has video output onboard, because I've never looked or tried. On boot it exposes itself as a network device over USB; you can SSH into it and start doing what you're trying to do.
What the RPI needs is a simple m.2 slot for a "real" hard drive, which are getting cheaper and cheaper.
a) write an OS image onto an SD card and just run off that if you want
b) set a flag in /boot. Upon reboot, the LEDs go into Cylon/Knight Rider mode while it copies the OS image from the SD card onto the eMMC. When they're done, pop out the SD card and reboot. Done!
Also, which OS version did you use? What Linux kernel version?
I'm more interested in longevity tests and industrial grade microSD card performance.
[1] https://www.jeffgeerling.com/blog/2019/raspberry-pi-4-needs-...
And everyone asks for longevity tests and industrial grade performance, but I always answer the same: I have had four Pis running almost continuously (99.9% uptime, occasional reboot for updates) for 3+ years, and have used the same set of 6 Evo+ cards for 3+ years now, and have never had any problems with corruption in any of those cards.
I also run a Kubernetes cluster with 4 Pis (see www.pidramble.com) from time to time (right now it's running), and I have never had a corruption issue either.
The main thing I attribute this to is the fact that I'm using reliable, quality power supplies (either Pi Foundation official supply or some other name brand or good PoE power supplies that support at least 2A).
https://www.amazon.com/Transcend-Wi-Fi-Class-Memory-TS32GWSD...
You can run custom code on it: http://dmitry.gr/?r=05.Projects&proj=15.%20Transcend%20WiFiS...
Personally, I used an Eye-Fi card when I needed wifi.
Also, microSD's form factor is probably too small to effectively house a reliable antenna, much less the additional WiFi chip. SD is probably the smallest feasible form factor today, and CFExpress and other cards are larger and might afford a little more space.
These cards work by inspecting the FAT32 filesystem on the card and separately connecting to wifi uploading the files on their own (so they work with any typical point-and-shoot or DSLR camera)
Having said that, what is so hard about running regular backup images, and simply burning a new one every time the current SD card inevitably fails? That's been working for me for years. Can't reboot... burn a new SD card from the backup. Reboot. All the super-timely stuff is on github, so at most, I might lose an hour or so every 6 months to a year. And anything that isn't altered often just has a rolling updated image for me to dd if needed. It's rare enough I just don't care about it.
I use rpis a lot, and since I work with MPI, usually there are quite a few running at all times. I just back up regularly and don't get alarmed that they don't last like a desktop at 100 times the price would. Of all the SD cards and actual rpis that died, I heaven't come close to that price, and have lost nearly nothing for all that I've gained in the experience.
The only one I would buy would be the RPI zero.
Having something reliable can matter.
From quick look at latest Linux source code, there are couple mentions of "blk_mq" in drivers/mmc, but there seems to be no actual support for multiple queues (unlike e.g. in drivers/nvme).
Does anyone know, if those are supposed to be same things? Does SD driver actually need new firmware to support multiple queues or is it something, that can be implemented on kernel side in software only? At the very least, it might be possible to use a third-party reader as long as it's kernel drivers are updated to take advantage of multiple command queues.
Unless I missed something, it's not that fancy. It's just one queue, not multiple queues -- the only new thing is that the length of the queue is more than 1. A multi-queue device is a device with more than one independent queue. This is useful for very high performance devices (NVMe, for example) that can allow multiple CPUs to submit requests at once without interfering with each other. If there's only a single queue, then all CPUs trying to submit need to contend for access to the queue.
read your specs, use what is recommended. not the mopst expensive. it's just engineering 101.
This isn't communicated well, the author points out that it's marketed as better, but you have to dig into the technical details to find out the host must support it as well.
Then some people dug deeper and found in the specs that 'Command Queue and Cache functions' needed to be supported in device firmware and/or on the kernel level for A2 specs to be reached—though I haven't yet found a way for any consumer to get their hands on devices or software with this support... so no way to achieve claimed A2 performance.
So then I bought an A1 card from the same manufacturer (SanDisk), and it is actually much faster than the A2 card (though for price/performance you're still better off buying the not-Application-Performance-class-rated Samsung Evo+ card).
And in the end, being able to use an SSD or mSATA drive would be even better (the former of which is semi-possible today with the Pi 4 and USB 3.0—though you can't fully boot without a microSD card yet).
Not true! The Pi supports several different USB and network boot modes.
https://www.raspberrypi.org/documentation/hardware/raspberry...
This documentation doesn't cover the Pi 4, but USB/network boot is available there too, cf.
https://www.raspberrypi.org/documentation/hardware/raspberry...
> Support for these additional bootmodes will be added in the future via optional bootloader updates
I believe they need to write (bootloader) drivers for the PCIe host and for the new USB3 chip before USB boot can be supported by the bootloader, while netboot requires a driver for the new gigabit ethernet core.
https://jamesachambers.com/raspberry-pi-4-usb-boot-config-gu...
No, they're not. Network booting is a couple of weeks out and USB over a month. It's a topic of discussion on the Pi forums.