Hacker News Reader

Comment by yakubin | Hacker News Reader

> NTFS is the de facto filesystem for portable data storage

I thought it was FAT32. Do Macs support NTFS?

EDIT: It seems they don't out of the box: <https://www.howtogeek.com/236055/HOW-TO-WRITE-TO-NTFS-DRIVES...>

EDIT 2: Since many comments replied with the same thing, I'll reply here: I wouldn't consider read-only support as support for portable data storage, since that allows you only to send data in one direction. When you want to copy a file from a Mac to a Windows computer, NTFS is not a solution now (out of the box at least), therefore falls pretty short wrt "portable".

View on HN

exFAT is probably better in terms of cross platform compatibility and addresses a lot of the limitations of FAT while remaining portable. Since Snow Leopard OS X is able to read and write out the box. Linux natively supported from kernel 5.4. Windows supports it from updated XP and onwards.

I not sure of Android compatibility though.

> addresses a lot of the limitations of FAT while remaining portable

Whilst at the same time missing a very important feature for portable storage - a journal.

Journaling can have significant downsides when used with external flash storage including reduced performance and increased wear of the flash memory.

Not disputing your facts, but I've used journalled file systems in flash drives since forever and never had a performance issue that was not directly tied to the hardware, and most have lasted many years.

I prefer having to replace a cheap flash drive more frequently than to lose data altogether.

Of course, everybody's use case is different and tradeoffs need to be considered.

Is exFAT's reputation for data loss/corruption unwarranted?

no, not journaled

I had an external harddrive with exfat that lost all its partition information.

Worked one day, then few days later plugged it in and it was all gone.

Just write all your data twice, lol.

If one of them becomes corrupt in a non-obvious way, how do you know which it is? You need three copies to be sure.

Or you could save a hash as well

What if a bit flips in your hash file and changes that? Now you need to store your hash a bunch of times too!

I kid, this is what I do. I've recently been backing up my family photos/videos to various media and I've been hashing each file and storing a hash file alongside the media to compare to in the future.

Not really. FAT32 is very simple and normally has a copy of each FAT table stored as well. When something goes wrong with the first FAT table, there's a reasonable chance you can still use the 2nd copy. [1]

ExFAT is much more complex and AFAICT it does not have a 2nd copy of all of its data. [2]

Anecdotal evidence is that I personally lost data on ExFAT file systems, but also have had many customers with problems when they figured to use ExFAT for my backup products. If I see ExFAT in a support bundle than -by default- my suggestion for fixing issues is to reformat to a different file system. More often than not, that's the end of the open support case.

[1] https://www.ntfs.com/fat-mirrow.htm

[2] https://www.ntfs.com/exfat-overview.htm

Android is annoyingly hit and miss. It's dependent on whether the device manufacturer wants to pay Microsoft royalty fees for it. So some devices do support it just fine and others don't. Often if a device has a microSD card slot and says it only supports 32GB cards it doesn't support exFAT since SDXC made the default file system for 32GB+ cards exFAT.

Microsoft exFAT patents expired in 2019, I think. It's included in the Linux kernel since 5.7 or some version around then. Android devices support exFAT depending on which Linux kernel version they have.

The patents didn’t expire in 2019, Microsoft just allowed Linux to use them without fear of getting sued.

And Microsoft ACTIVELY went against Android manufacturers to get licensing fees a decade or so back, so those that didn't want to pay are much more likely to have a hard ban on exfat in place that will take time to overcome even after Microsoft opens it

Maybe that will change in the future but most Android devices are not using a 5.x kernel yet. All my Samsung devices support exFAT and use Android 11 with a 4.14 kernel. My old BB Priv also supported exFAT with Marshmellow which definitely did not have kernel version 5!

Just recently I brought a Mac-formatted exfat flash drive to a print shop, where it failed at being read on a Windows box. I doubt it that they're using WinME.

It might be an issue with partitioning. ExFAT support built in from Windows 7 and requires service packs for earlier versions.

exFAT is HORRIBLE. Sorry for the caps. Interoperabiliy between Windows and Mac is hit-or-miss. Formatting on Windows and it's random if Mac can read it. Unplug without ejecting and expect to wait hours for Mac to analyse and fix, or plug-in and out and in and out of Windows and hope it fixes it. Many times a day. Zero problems with NTFS.

Mac does not support NTFS writing out of the box. This alone makes NTFS not suitable as an universal format.

Try UDF.

> exFAT is probably better in terms of cross platform compatibility and addresses a lot of the limitations of FAT while remaining portable.

Still it is a shame that we're kept back literally decades when it comes to file exchange due to corporate culture polluting every corner of IT. Ensuring file export options plus file formats full documentation, should be among the most important requirements any software should satisfy to be even considered for professional use.

I agree with your want. Wholeheartedly.

Alas you are right in that this culture is all over Corporate World (yes not just America). Vendor lock-in is something that's been with us since forever and I doubt it will ever go away. They might have gotten better at hiding it from obviously being spotted but that's about it and in some other cases it's still just as clear as always but the marketing people are still very good at selling it to customers.

OS X/macOS used to have some serious issues with exFAT but they may have been addressed in the few years since I last used exFAT on a Mac. Something to double-check before using the pair to store critical data.

UDF is the "secret" multiplatform FS there. Even under OpenBSD you can use udftools and write pretty fast to any external USB storage.

Oh I forgot about UDF. Easy to forget since the GUIs dont usually make it an option when formatting.

Unsure about Android and ChromeOS compatibility.

FAT32 doesn't support files bigger than 4GiB, that's a huge limitation these days. But I don't disagree that calling NTFS "portable" is a bit of a stretch, although hopefully that'll change if good quality open source drivers become available.

NTFS has journaling,compression, encryption, max 255 character filenames, max partition size is hundred of TB rather than 2GB, generally faster performance wise.

Trouble is , FAT32 has more OS compatibility.

Would be nice if Linux did have first class support for NTFS as it’s superior. But right now for maximum compatibility you need FAT32

> max 255 character filenames

it doesn't in spec, that's just the win32 API

>it doesn't in spec, that's just the win32 API

NTFS filename length is limited to 255 Unicode chars.

You may be thinking of full path lengths of 32,767 Unicode chars but the filename component is still limited to 255.

If you look at the NTFS data structure "$FILE_NAME (0x30)" at byte offset 0x40: https://www.google.com/search?q=ntfs+%22%24FILE_NAME%22+byte

... there's only 1 byte used to describe the filename length. Hence max of 255 chars.

Wikipedia page about NTFS also mentions 255 max length for filename: https://en.wikipedia.org/wiki/NTFS

You're confusing the filename length limit with the limitation on the entire path length in the Windows API. NTFS allows paths to be up to 32k characters long, but each path component tops out at 255 characters.

The Paragon driver cannot recognize encrypted NTFS volumes AFAIK

> max partition size is hundred of TB rather than 2GB

FAT16 is limited to 2GB, FAT32 can extend to 2TB (but the max file size for a single file is 4GB, as the parent stated)

A bit of personal experience: I've needed to create a Windows 10 installer from a MacOS computer a few weeks ago. Without the MS helper program I've figured I should perform the following steps:

1- download iso

2- format usb with fat32

3- copy the data

Apparently you can't just dd the ISO file into the USB block device like every other Linux or BSD ISO. It need to be FAT32 to be able to boot through UEFI.

The issue however is that the .wim file containing the OS installation source is bigger than 4GB. I think its sources.wim but I'm not sure. I had to install some command line utility from brew that did the job of "optimizing" the WIM file and managed to reduce it to a file bellow 4GB.

What does optimizing means? Better compression? Cleaning up crap inside the image? If their ISO tool probably does this without user interaction then why include a file bigger than what the FS can handle in the first place?

I don't know if this counts as some kind of dark pattern or MS being really antiquated in terms of bootable flash drivers.

It's a case of it working fine on Windows, so, there's no need to bother supporting anything else.

What I'm doing is just using GPT partitioning, creating one NTFS partition and copy all files there. All my computers were able to boot in UEFI mode from this USB drive and successfully install Windows. With Linux it was a bit more cumbersome, but most Linux distros are hybrid ISOs anyway, so it's not as big issue. I know that UEFI BIOS is not obligated to support NTFS partitions, but my little practice they do.

Good to know that this is an option but from MacOS I don't have NTFS write capabilities. I think ExFat is available so I might try two partitions one being the EFI ESP and the second containing the rest of the install files.

It's actually exFAT thats portable. NTFS is mostly for windows users, and macs do support it - though only reading.

Anyone know why? Given ext2 has had a public domain implementation for more than a decade? Is it because windows & mac won't support ext2 even for free? Seems like a better lowest common denominator fs than any of the fats?

You can't use easily ext2 on an external storage because of permissions.

Isn't that how it's mounted that matters there?

I never find an option to ignore permissions.

That's because Linux and OSX ignore NTFS permissions making it work more like FAT.

Plug an existing NTFS drive from one Windows system into another Windows system and you get a similar issue where opening a home directory requires you to take ownership.

But Linux refuse to ignore permissions on an ext2.

that depends, exFAT lacks permissions and symlinks. I think FreeBSD has NTFS support but lacks exFAT

> that depends, exFAT lacks permissions and symlinks.

The lack of permissions is why it works as a portable file system. If you've ever tried to use, say, ext for this it becomes clear why nobody does that.

You can just pass -o uid=X,gid=Y when mounting ext2 to solve this.

I have no idea if I did something, but after MacOS 11 upgrade it seems to automatically mount my NTFS drive.

I do know that FUSE MacOS stuff seems to be disabled in brew.

This has been true for a long time. Does it mount it r/w or read only?

Interesting. Previously I had to run the mounting command manually ever since I remember trying to mount an NTFS drive.

It's read only.

IN my mind it has been exFAT for the past several years.

Buy an external hard drive or USB stick, unless it's marketed as "for Mac", it's going to ship formatted as NTFS.

memory cards are FAT32

IME most external hard drives and USB sticks are marketed as compatible with both Windows and Mac, and ship formatted with FAT32. But maybe I'm the odd one here.

Most these days ship as exFAT.

Literally NONE of the 100s of Tb I have purchased over the past year in formats ranging from SD card to USB stick to SATA drives have been formatted as NTFS. None of them.

Macs support NTFS in read-only mode out of the box. For writing you need tools like Paragon NTFS.

They do, but only to read, not write. I think the current up to date recommendation for cross-OS compatibility on external drives is to use ExFAT, though I believe the concept of the ‘external drive’ itself should be deprecated.

> though I believe the concept of the ‘external drive’ itself should be deprecated

why?

I think they’re problematic because they’ve reached ungodly amounts of storage and I don’t think most non-technical people fully understand how much of a loaded gun a non-regularly-backed-up 2TB external HDD is.

As a real world example, I’ve had to help recover my aunt’s 5-year-old 2TB HDD that she dropped several times and had no backup of, and by helping I mean confirming she probably lost a terabyte plus of data unless she wants to pay five digits USD to pay to a recovery company for a way less than 100% recovery. She is a highly educated professional (a MD) and yet, it did not even occur to her that the drive will not only might fail, but rather, will fail.

To quite a lot of non-technical people an external drive appears to be the safest form of storage, considering its being physically visible and movable. The closest analogy is storing money under the pillow versus a in bank, in all the good and bad ways.

Yeah, you need to use FAT32 if you want to make a cross os portable drive.

Reply on news.ycombinator.com