The thinking behind the 32GB Windows Format limit on FAT32
theregister.com
theregister.com
Changes, especially in a large system with a large user base, require extensive testing, and even small changes may come with a lot of red tape. I don't know how things work at Microsoft, but I wouldn't be surprised if making changes to the UI for a technology that's being superseded anyway just didn't make anybody's list.
Therefore FAT32 will remain preferred until the patents expire on exFAT, even though the specification for exFAT has been released, things like that have proven to be not good enough until after the patents expire.
The purpose of exFAT appears mainly to be an attempt to foist a new encumbered filesystem on developers now that the first mainstream Microsoft filesystem is in the public domain.
We will know when Microsoft has reduced their anti-user stance on filesystems after they have released very detailed full NTFS specifications in the most useful way possible.
Also I have the greatest respect for Dave, and like he mentioned he was only doing the UI not the bitmoving of the actual Format process.
Those who were developing the NT version of FAT32 formatting failed in much more serious ways where Dave did not.
The inconsistency compared to FAT32 formatted by W9x is actually intolerable.
Best reliability still requires booting to the DOS from W98SE to properly format FAT32.
They are not useful for anything else except memory cards, USB sticks and external SSDs.
Unfortunately, nowadays it is frequently necessary to copy files larger than 4 GB. When using FAT32, you cannot do that without splitting the file and concatenating the parts at destination. That wastes time and it is annoying and sometimes it may be even impossible, when the destination lacks a concatenation program or an archiver that can deal with split archives.
After Microsoft released it, now exFAT is supported in the Linux kernel. Because exFAT is also supported by all commercial operating systems, it is now the best way for file transfer.
It does not matter any more if there are still valid patents for exFAT.
Would you care to elaborate for why that is? Would anyone else with knowledge about the patent system and any possible risks imposed by it also like to do so?
IANAL - so there's likely some caveats (you probably can't just use the exFAT name unless certain qualifications are met, just like you can't use Firefox's trademarks willy-nilly), but that should be good enough for just about anyone. Including paranoid OSS peeps like me :)
0: https://cloudblogs.microsoft.com/opensource/2019/08/28/exfat...
No its inclusion in Linux was the result of a decision by Microsoft, not a granting of an open license. Any other implementation still has to apply for a license:
"For customers wanting only a patent license, you can contact us to learn more about the exFAT licensing program and request a pricing proposal."
I don't know how the release to Open Invention Network affects things, though.
To put it bluntly, if Microsoft's guarantees are good enough for Linus so as to merge the driver in the Linux mainline, they're good enough for me too.
But for a fun fact, if you see many embedded systems that say they can only support an SD card up to 32 GB, it is a soft limitation due to the fact that media over 32 GB are pre-formatted to exFAT (and the system lacks support for exFAT). FAT32 can go up to 2 TB. So I have found that reformatting a larger media back to FAT32 makes it recognizable to that system with all of the storage space.
There might also be the issue of the card reading hardware only supprting SDHC instead of SDXC (or SDUC).
My old laptop has Ricoh SDHC card reader. Windows can't read 64GB SD cards (properly formatted with exFAT according to the standard) on it because it's using the proprietary Ricoh driver.
On Linux it works just fine because it's using the generic SD card reader driver which supports SDXC.
You seem to be very optimistic about how fast these get implemented into embedded devices.
OTOH, I do have the Windows logo on my steering wheel, and somewhere inside the car there's a Windows CE system installed (it doesn't have a multi-color LCD screen and interacts using speech recognition, so it's not really obvious).
The spec doesn't mention backwards compat, but does mention as a goal
>Retain the simplicity of FAT-based file systems.
I would like to test out if an input embed system is exFAT compat. And what might be required to update.
SD under 32GB uses FAT32 so you can’t drop supports, and FAT32 stack is often already pushing the maximum ROM size, so no room is left for a novel filesystem. Lots of developers aren’t themselves proficient at writing filesystem code as well, and there aren’t as many copy-pastable exFAT implementations as there are for FAT32.
So unless they know what they’re doing and unless it’s critical to their product, the majority won’t easily move onto exFAT.
As for the use within the device “firmware”, the user defined program is usually just a giant baremetal a.out in the middle of the ROM at a specified address, so no FS is involved there.
I have had issues with ntfs getting corrupted and losing file both in mac and linux at times so there's that risk with implementation too.
Has there been any changes to these since MS has opened the license to exFAT?
Regarding Linux, exfat-utils only provided limited functionality. The kernel mainline lacked a native exFAT driver. Mounting exFAT volumes was possible only through FUSE (with all its shortcomings).
Therefore, if you have a recent Linux kernel, you no longer need to install a user-mode driver for exFAT and the kernel driver has better performance.
[0]: https://kernelnewbies.org/Linux_5.7#New_exFAT_file_system
It's a bit of a shame Microsoft didn't push and open up exFAT earlier and allowed FAT32 to become an issue again, only with removable flash drives and memory cards this time.
[0] Non-native english speakers may be confused by this: https://www.merriam-webster.com/dictionary/as%20is%20someone...
Erm, because that was the idea? SDXC is a big racket and Microsoft has been collecting patent fees from any kind of consumer device that's capable of reading memory cards thanks to the filesystem requirements in the standard.
The article states that you can format as fat32 from the command line - this wasn't my experience. Disk Util (cli) saw my raw partition and started formatting it as FAT32, but seemed to fail about an hour in saying the disk was too big.
I ended up finding a random abandoned GUI utility which seemed to work, though it's extremely infuriating.
This can also really hurt in unexpected ways with otherwise badly written drivers. Nintendo's driver for the switch would constantly write to the table unnecessarily, and while this doesn't do all that much in normal operation (other than waste a small number of writes), it made it drastically more likely (in some cases for some games almost guaranteed) that the table would be partially written and corrupt if the system crashed or lost power unexpectedly. When some new Pokemon games came out and were initially a bit unstable this came up a lot.
That day might be soon
[0]: https://lore.kernel.org/linux-fsdevel/2911ac5cd20b46e397be50...
On the other hand I've been a heavy user of exFat formatted sd cards on Samsung phones for long years and had no problems with it, so hopefully the new Linux driver (which Samsung upstreamed) will be solid.
One example: FAT32Format GUI - https://www.softpedia.com/get/System/Hard-Disk-Utils/FAT32fo...
Instead the code sat untouched for decades as the limit slowly became more and more unreasonable.
IMHO the real issue is the artificial limit. Instead of straight up disallowing you from creating large filesystems, it should have had a dialog box warning about small files being inefficient but letting you make it anyway.
My projects likely won't span decades but when I know I'm putting in some opinionated value I leave behind a comment like:
// XXX picking 32gb because it'll probably be 5 years until we have >32gb hard drives. Change this when the world changes!
But we have to admit that storage has leapt hugely in size and very rapidly. In 1990 or thereabouts, a common hard disk size was about 30 megabytes. Today I have files that are many times that size, and we think that that is quite normal. A mere five or so years later around 1996, that size had jumped to 4000 megabytes! These days my laptop has 6 million megabytes.
We can't blame filesystem planners for lack of foresight. They were blind-sided by progress just like the rest of us were.
This wasn't a lack of foresight, this was being prudent. If I can't come close to testing an upper limit, it's correct to acknowledge it may degrade in a way I haven't anticipated.
Especially since he was writing a GUI which should generally avoid handing users footguns, and which can be updated over time.
Also why not tar a drive for archive if I want :}
Is it just that they are different formats, and exFAT has taken over, in terms of pre-formatted USB media?
So maybe his work was on Windows 2000?