SD Memory Card Formatter for Linux | Hacker News Reader

SD Memory Card Formatter for Linux

sdcard.org

41 pointsW4phle_Stomp59 comments

18 months ago I whined to their devs it'd be nice to have a version of their SD formatter for Linux, et voilà!, 18 months later, BOOM! - Awesome!! ... I"m not sure if my petulant whining and life-draining email verbosity provided the underlying motivation to their acquiescence, but until I'm provided contrary proof I'll assume it's so ;o)

Thank you! Thank the devs more of course ;)

My world depends on writing TBs of data to micro SD cards in difficult environments right at the limit of write speed. These tools are great but our devices are linux (embedded), so this really is a huge help.

For anyone else out there who needs high performance out of an SD card... fstrim is your friend. Seriously, nothing comes close. Run it regularly when not under load.

> Developed by Tuxera

https://www.tuxera.com/

I've always formatted SD cards like any other block device. This page mentions a secure section of the card that can somehow reduce performance if you don't treat it specially? What is this security section for, and do standard UNIXy partitioning tools have the ability to handle them properly, or do I have to use this proprietary(?) tarball if I want good SD performance?

This is really the first time I hear about special tools required for SD card formatting. Is it also “wrong” to use MacOS disk utility to format SD cards?

IIRC there is a reserved area in the beginning of th SD cards flash which is used for internal housekeeping. When this area is overwritten/formatted the card is bricked. [add] The size depends on the size of the card. I think it is around 32KB on cards with a capacity under 2GB.

I do some development with dev boards and firmware stuff and regularly dd images onto SD-card without any skips or something like that. I never bricked any card.

I'm dubious about this claim, I have completely destroyed the partition table and mkfs on the raw device...

Unless that was an mtd device then you weren't really writing to the raw device

It did show up as sda.. can mtd devices show up as MTD, i thought it had to use some "MTD" app to transfer files to it.

As I understand it there is a controller on your device that emulates a disk drive. The controller is accessing the raw device on your behalf and _should_ not be overwriting any sensitive/forbidden areas of the flash that might brick the device.

You might be thinking of MTP, used by eg. phones that don't present a block device over USB so that they can internally use a filesystem not supported by Windows; MTP is a higher-level interface more akin to a network file system protocol.

mtd is a Linux driver subsystem for accessing raw NAND or NOR flash devices that don't have an SSD's Flash Translation Layer (FTL) in front of them to provide features like wear leveling. It's most commonly encountered in embedded systems.

You are correct, I was thinking of MTP.

Thank you for the education.

You're right but it doesn't matter here as it is fully hidden. Unless you have an SD card with a buggy firmware or access to manufacturer-specific tool, it is not accessible even when you "deep" format it.

The reserved areas are managed by the ARM cores running inside the SD cards. Your host machine doesn't see them.

In some SD cards you can use proprietary magical commands to communicate with the firmware and update it or ask it to mess with the hidden sections for you, but those options aren't exposed to any disk copy tool I know of.

I don't think the page actually says that the security section has anything to do with the potential for poor performance; those are discussed in separate paragraphs. It's mentioning the security section to warn the user that this tool will not actually wipe every byte of the card, just the normal unsecured data storage area.

The performance pitfalls are not stated, but I'd guess they're referring to the possibility of using suboptimal file system block sizes or alignment. I wouldn't worry too much about recent tools getting alignment wrong or recent cards being overly sensitive to block sizes.

The official SD tool probably also ensures that it uses the exact filesystem called for by their standards (ie. when to use FAT32 and when to use exFAT), as a guarantee of compatibility with any device that interprets the standards too strictly.

I can't speak to the relevance or necessity of all this but I have adapted a script from this blog post[0] when formatting my Raspberry Pi's. If I recall correctly GNU Parted's manual also mentions a 4MiB offset and both link to a common article[1].

[0]: https://web.archive.org/web/20230405090708/https://3gfp.com/... [1]: https://web.archive.org/web/20230406150117/https://lwn.net/A...

I call bullshit, if performance was an issue the companies that make SD cards would have found a way to fix this in the firmware and not have to rely on an external formatting tool when almost every device with an SD slot has the ability to format an SD card.

This is what the S in SD stands for - "Secure Digital". SmartSD cards - https://en.wikipedia.org/wiki/SD_card#smartSD_cards are actually the norm, because the simple spec is easy enough to add to the flash memory controller.

I know someone who worked on a weird SD-Card based "Video Rental" service[1][2] - The idea was that you'd bring your standard consumer SD card to a redbox-like kiosk, load some video onto it, and then be able to play it on their special but standard (integrated into smart tvs by Toshiba et al) video player, which would automatically self-destruct the encryption keys using that secure element.

[1] https://www.engadget.com/2010-12-08-mod-systems-download2go-...

[2] The far more interesting product of this company was all of the weird fraud that their founder was engaged in. https://www.geekwire.com/tag/mod-systems/

Late April fools??? "Protected area" but not protected against formatting? (Unspecified) performance issues? But each and every camera, OS, or other device will happily format and use SD cards w/o problems? Sounds VERY strange to me...

Console hacking forum users such as gbatemp stress the use of "Panasonic SD formatter" when formatting SD cards as FAT32 and guiformat for exFAT, I don't know if there is a reason behind this advice or if it's just cargo culting.

>I don't know if there is a reason behind this advice or if it's just cargo culting.

Easy to reproduce results is probably what they're going for. If that tool doesn't have much configuration possible, many footguns are prevented.

My experience is that both of these tools do something for aligning partitions or file systems that a regular right click+format won't do. I've had SD cards fail to boot in a Raspberry Pi because of these small details.

I don't think they do anything that other formatting tools can't do, but they do make the alignment much easier by defaulting to the right options.

So much confusion in this discussion. Let me try to clarify (all of this is very very simplified).

1. NAND in SD cards has page size (minimum writeable piece) something like 4kB or 8KB, or maybe even 64KB. But as per spec SD cards must allow writes at 512 byte granularity. Aligning your file system clusters with these flash pages will allow SIGNIFICANTLY better performance and lower wear. How? Well in the CSD register (accessible using SD commands), the NAND geometry is given, so all it takes is reading that and creating a file system that is properly aligned.

2. Why not just mkfs.vfat? You can, but unless you do some of the above work yourself and some math, you’re unlikely to get the alignment right and will suffer because of it, especially on small or random writes. You CAN do it yourself. Just takes a little work.

3. What about my gopro, my phone, or $DEVICE-X that can format SD cards? Well, most device firmwares DO indeed do this correctly as the SD spec advises to do.

4. Why doesn’t the card just do it in firmware all by itself? Curiously sony memory stick does this. Why not sd? There isn’t a command in the spec for that and adding it now is too late - nobody will use it. Thus we have this tool and ones like it.

5. What is this about the secure area? The tool just says it doesn’t touch it. Almost nobody does. Costs a lot of money to license the SD security stuff. Nobody uses it. It was meant for securing MP3s on cards. Secure area exists on all cards (yes you are paying for flash you cannot use). Luckily it isn’t big. Special commands are used to access it. This tool and basically any device you’ll ever encounter in your everyday life do not use them. Lacking those commands, it is as if the secure area doesn’t exist.

6. Why dont windows/Linux/macOS do it right themselves? My guess? Separation of concerns. The formatting code there only formats a partition. For proper alignment you may need to start the partition on the right block as well. Simple example. Say in your fictional card, we have 4KB pages. That’s 8 512-byte sectors. So we want our first FAT cluster to start at a multiple of 8 (and be a multiple of sectors in size). Say we work out that our various FS Structures need 17 sectors. That means that we need the partition to start at sector 7. Thus we make the MBR do so. Wait! You might say that FAT already has “resevered sectors” value we can use for this. No MBR touching needed. Well, you’d be surprised how many devices/firmwares break with nonstandard values there. Also, what makes you so sure that mkfs.vfat can even find out how many sectors into /dev/sdb, /dev/sdb1 starts, to do the math right? For that matter, what makes you think it would even know to check for SD-card-ness of /dev/sdb?

> Well in the CSD register (accessible using SD commands), the NAND geometry is given

Is this difficult to obtain (e.g., via ioctl(2)) and parse?

[edit] looks like it's readable from sysfs: https://www.kernel.org/doc/html/latest/driver-api/mmc/mmc-de...

Parsing is another matter. It would be really useful if e.g., parted could read/parse that and then automatically align created partitions!

Partition alignment also depends on the file system options you plan to use to format it. Eg: depending on cluster size you choose, the size of the FAT differs, which in turn shifts the start sector of the first cluster. So, really, partitioning and formatting NEED to be done cooperatively and not independently. That’s the purpose of this tool, really.

Yeah good point. Back in the distant past, parted manged the creation of filesystems as well as the partitions on which they are created. It's a shame that was all removed because this is exactly why it's needed!

How is this better than dd(1)? Does it "format" the card with any specific filesystem or just overwrite/erase its contents?

The answer is in the third paragraph.

> SD/SDHC/SDXC Cards have a “Protected Area” for SD Card security purposes. The SD Memory Card Formatter does not format the protected area in the SD/SDHC/SDXC Cards. The protected area shall be formatted by an appropriate PC application or SD host devices that provide SD security function.

If this "protected area" even exists [1], both dd(1) and this thing seem not to touch it. So no difference here.

[1] Never heard of it and I've erased and recreated partition tables on SD cards multiple times with no "bricking"; maybe it's a feature of cards supplied with "encryption software"?

From [1]:

> Content Protection for Recordable Media (CPRM), the content protection technology used for SD standards, is key to enabling a distribution system for music, video, etc. CPRM provides a high level of protection against illegal copying.

Sounds like someone intended to sell movies on SD cards, replacing DVDs/bluerays.

[1] https://electronics.stackexchange.com/a/316421

Ok. So this area has to be accessed by some special commands and is not exposed as part of a block device.

The "SD Memory Card Formatter for Linux" mentions this non sequitur seemingly to confuse users and cause FUD.

I've linked elsewhere: https://www.engadget.com/2010-12-08-mod-systems-download2go-...

Mod Systems was a Toshiba-funded "Video Rental on SD Cards" startup, whose primary product was a set of bizarre fraud cases.

I fear it's closed source?

It may be proprietary but it's developed by Tuxera, the same lads who created and maintain the NTFS drivers for Linux.

So what? Don't get ne wrong, tuxera is cool but this still does not make it free software.

If it is a choice between preparing my sd cards with open partitioning tools, or running some non-auditable official tool in binary form, it is an easy choice.

I'll do the former, as I always have.

By the way, a cool quote from the agreement preceding the download page:

>3. RESTRICTIONS: You agree to NOT: (a) disassemble, reverse engineer, decompile, or otherwise attempt to derive any source code for the SDA Software from executable code;

This should be quite alarming. They insist that we format SD cards with their tool, but they don't want us to know what the tool does.

I will not be the one reversing them, but I checked it is possible to download these through tor, and recommend downloading them that way to prevent potential trouble.

The hashes for the files I obtained are:

SDCardFormatterv1.0.2_Linux_ARM64

MD5: 2CF66295A29C5F496A489074CBC42CFA

SHA1: EE06C0C6834EB9F53E66343A7CFDB95FCCB5FF66

SHA256: 8695A0F441129845136AFEE01D94D6DBCCBE87BA7C904F045F2AF0613F9A1978

SDCardFormatterv1.0.2_Linux_x86_64

MD5: 459A698C3961BACA8103173B3CDA7173

SHA1: C28EFFDF05FF186E56C57476C2794CC7A7AE3AF0

SHA256: 3D961085954ABEB764265184A92C1114AA4CEF9CF12CA3C3337B6B63E0DBE0FB

> 3. RESTRICTIONS: You agree to NOT: (a) disassemble, reverse engineer, decompile, or otherwise attempt to derive any source code for the SDA Software from executable code;

Depending on where you live, this might have no legal effect. Some jurisdictions allow explicitly for reverse engineering & co. for different purposes (e.g. research, interoperability with other systems, etc.).

SD Cards have computers inside of them. Security conscious users will be really worried using a blob for their stuff.

Hard drives, USB sticks and SSDs too...

Are there any open source sd card firmwares?

Would be an interesting project, I have no idea how you could reprogram them though.

Some manufacturers have proprietary undocumented APIs that allow the host to upgrade the firmware in the card, but I’m not sure every maker does, and very sure most wouldn’t bother to update their firmware unless some horrendous issue came up.

Is closed source not allowed for Linux?

If you want an open source SD card formatter for Linux, you can mkfs.vfat /dev/mmcblk0 all day long, works just fine.

It's allowed but people who use Linux want to know about these things ahead of time, as many of us seek to reduce the amount of closed-sourced code running on our systems to give us peace of mind.

Would you entrust a software you can’t audit to format your computer’s disks?

People who use Windows or Mac do that all the time?

Would you entrust a software you can’t audit to format a device with software you can’t audit?

It'd be great if somebody with knowledge about this could make a benchmark comparing the performance after using this program versus using the standard tools like dd, mkfs, etc., using different parameters for things like the alignment (if that's possible at all using that tools).

Hidden protected area in media + closed source software that writes it = potential tracking danger. I'm not implying anything, but putting those two things together could be dangerous.

Of course that should apply to all proprietary software accessing SD cards, therefore all phones, all cameras, etc; thankfully, being among the least reliable media out there for carrying files, and less practical than for example a USB connection or dongle, they're often restricted to the same hardware using them, as would be any potential tracking data.

Hmm… I recently bought a modern original PlayStation memory card that saves to SD.

The first SD card I bought just would not work with it no matter what I tried.

Support actually had me download the “SD Memory Card Formatter for Windows” from this site and reformat my card. Did not fix the issue. I ended up just buying a new SD Card which did fix the issue.

At the time it just felt like support was grasping at straws and wasting my time, but sounds like maybe not? Very curious. I have always just formatted my cards with Disk Utility in macOS.

You may have purchased an SD card with a fake/spoofed capacity, which can cause all sorts of issues. This is apparently a pretty big issue with stores like Amazon that comingle inventories from multiple vendors.

I would bet money it was a Microcenter store branded one, the ones they have at the checkout counter. They are what I almost always use and they have never given me issues before. I am also sure I have thrown it into something else at this point though it is difficult to say what.

Maybe the memory card uses SPI to communicate with the SD card? The SD card standard mandates SPI support, but some SD cards are non-compliant.

Reverse engineering what makes this tool special sounds like a fun weekend project :)

But to download it you have to agree to not do exactly that :(

I am betting this came out because of Valve steam deck popularity. A linux based device that uses sdcards

Not many smartphones or cameras run Windows either, if this being for Linux is what you're talking about. I'm also not sure why they'd bother with multiple cpu architectures in such a case. Sdcards are so widespread, I really don't think Steamdeck is what made them think they might need a raspberry pi, arch linux, and debian 10(!) release.

Huh? I've always used dd and the performance seems quite acceptable.

Is "format_sd" going to find its way into all the Linux distributions as a standard command? Would it's license preclude doing so?

>Would it's license preclude doing so?

It would. Most distributions will never ship this, because it is closed source code of questionable usefulness.

Reply on news.ycombinator.com