truly an appropriate name
In practice, UTF-16 validation is rare.
https://learn.microsoft.com/en-us/windows/win32/intl/surroga...
NTFS, on the other hand, has no dictated format for filename encoding. It just stores raw bytes as filenames, so anything can be a filename on NTFS, including invalid strings if the caller decides to do so. That's not on Windows either, otherwise, we should add Linux to the list too as ext4 and most other file systems also don't care about filename encoding.
It just happens that most Windows programs will treat it as UTF-16 and it'll mostly be fine. Until you run into the edge case where you do have file names that aren't UTF-16.
This is basically a result of Windows's file API being fixed in 1993, a good few years before Unicode decided that 16 bits wasn't enough for everyone, and hence a good few years before UTF-16 existed.
A file path is just a sequence of bytes. Don't try to validate it in any sort of encoding, it's just bytes that the OS uses to identify a file.
"That's how unix does it as well" is the exact same thing. Most people these days expect file names in UTF-8, but not all file systems restrict the valid character sequences to be UTF-8 compliant. If you treat everything as if it must be UTF-8, your application may just break when it encounters a name that isn't UTF-8. (This detail gets messy because there are configurations of, eg, ext4 and ZFS that limit valid file names to be valid UTF-8.)
On Linux, it’s not as rare as you might expect to end up with a few non-UTF–8 file names, though they’ll normally be on things you won’t often touch directly, so things like desktop software can reasonably only support UTF-8 paths.
I'm getting strong XKCD "Workflow"[0] vibes out of the idea that someone would have unintelligible paths on their system
I have quite a number of times. There's no one "thing"; it's usually some legacy app that write these names. And when you get a program that refuses to open any non-UTF16 compliant file name, you'll have a time of grief.
Beyond that, before the mid-2000s, it was common to use non-UTF-8 locales on Linux. So I'm sure I still have ISO-8859-1/—15 encoded file names somewhere, especially in archived data. They're not always trivial to rename either, because there might be references to them by name. (Or, in odd cases, you can't convert the name to UTF-8 because you hit a filename length limit, since UTF-8 is more bytes).
I believe wanting to access data from 20 years ago is a perfectly reasonable use case.
It's not so bad if a program can't display the file name right, as long as it doesn't crash with an exception or refuse to open the file. Unix file names have been defined as arbitrary sequences of octects except / and NUL for 30+ years.
That works fine until it becomes literally impossible when your language/library/framework forces you into picking an encoding.
Even Microsoft's own APIs that add "UTF-8 support" choke when they encounter file names with invalid UTF-16 sequences.
You can't just dismiss the whole operating system API space just because NTFS doesn't enforce filename encoding. Yes, you might encounter file names with invalid UTF sequences because some app might be allowing them to be created. That can also happen in Linux too. Then add Linux, and probably others to the list. Then, what's the use for this list?