Hacker News Reader

Comment by mcculley | Hacker News Reader

This is interesting. I was just recently working on an app where I wanted to ensure the UI wouldn't accept problematic characters in filenames. Obviously, Unix has problems with '/'. I'll add ':' to the list. That's unfortunate. What else should portable apps avoid?

View on HN

Microsoft seems to have a fairly comprehensive list:

https://msdn.microsoft.com/en-us/library/aa365247(VS.85).asp...

They suggest avoiding <>:"/\|?* as well as all ASCII characters 0-31.

ASCII 0 can be really fun. Lots of filesystem APIs deal with NUL-terminated strings (like, all of POSIX) so a zero byte in the middle of your string just truncates it at that point. If you use something that tolerates zero bytes for your UI strings (like NSString on the Mac, maybe C++ UI frameworks dealing with std::string) then the full string may show in the UI and you just mysteriously get a filename that's shorter on disk than what you see on screen.

ASCII 255 used to be fun in the Windows 3.1 days. DOS would handle just fine (displaying whitespace). The Windows Explorer (or whatever it was called back then) would not let you select a directory named like that. Basically this made a directory inaccessible, unless dealing with very tech savvy people.

If you want a serious rabbit hole, think about Unicode characters in filenames. Windows filenames are supposedly UTF-16, but they do not enforce the requirement that surrogate pairs (which represent characters over 0xFFFF, like emoji) must actually be paired, so you're not guaranteed that a UTF-16 decoder will actually read a filename successfully. UNIX filenames are just untyped byte strings that don't contain NUL or /, which by convention are ASCII or UTF-8 these days, but nothing enforces that; if you run ls, it will just print whatever bytes are in the filename to the terminal, and make it the terminal's problem. So round-tripping an arbitrarily weird UNIX filename to Windows, or vice versa, is challenging.

If you're in a position to enforce well-formed Unicode on all platforms, you're much better off. But many things (e.g. backup systems) don't get the option to just refuse files they don't like.

This points to a fascinating underlying difference between the operating systems. In general, UNIX attempts to be completely non-character-set aware and is built with the philosophy that how to render characters is strictly the terminal's problem. Windows, on the other hand, has a notion of a system character encoding and will try to keep everything compliant with it (with mixed success).

There is a very important takeaway of this: case-sensitivity. UNIX cannot be case-insensitive for file names because the mapping of lowercase to uppercase characters is dependent on the character encoding used, which it doesn't know. Windows can (and does) coalesce case for file names because it knows the character set in use and can consult the relevant mapping.

This difference in behavior produces all sorts of frustrating behavior when interacting between the two platforms, e.g. the classic case of Windows SMB mounting a share from a nix server that contains two files differentiated only by case. It'll show both entries but think they both point to the same thing. On the other hand, it's easy to create file names on a Windows device that are near impossible to name on nix. These are important things to be aware of if you ever implement a cross-platform network user environment.

> Windows can (and does) coalesce case for file names because it knows the character set in use and can consult the relevant mapping.

Actually, it's not the character set in use. Windows uses a case mapping table which is part of the NTFS filesystem metadata. See for instance https://web.archive.org/web/20110308034840/http://blogs.msdn...

(Yes, this means that the mapping of lowercase to uppercase characters can change if the file is copied to another drive in the same machine!)

There's WTF-8 to convert broken UTF-16 into something UTF-8ish. https://simonsapin.github.io/wtf-8/

I kind of wonder if paths not being allowed to contain NUL or '/' was one reason why for codepoints that are represented through more than one byte in UTF-8 (-> all non ASCII codepoints) all bytes have the most significant bit set to 1 (https://en.wikipedia.org/wiki/UTF-8#Description) This makes it impossible to have multi-byte to contain valid ascii chars like `/`.

Note that macOS actually does decomposing unicode normalisation on file names, I guess because it makes handling case-insensitivity easier. (Just doing ascii case insensitivity also handles o+diaresis, but not the ö codepoint) https://developer.apple.com/library/mac/qa/qa1235/_index.htm...

This is cribbing from source of a filename sanitizer in one of my company's internal libraries. The function is a little... paranoid... so I'm not positive all of these are actually forbidden.

/ and 0x00 for unix

:?"<>/|\* and chars 0x00 .. 0x31 for windows

'~!#$&%^; if there's a chance of filename being passed to shell w/o proper escaping.

Windows also forbids a bunch of filenames matching regex "CON|AUX|PRN|NUL|COM[1-9]|LPT[1-9]"

Also, ending filenames with space or period really messes up windows. File explorer can see it, but can't delete or rename it.

edit: fixed markup

That should be NUL (one L). Interestingly, when I tried it in Powershell, 'type NUL' reports that the file does not exist, but in CMD, 'type NUL' outputs nothing (it's the DOS equivalent of UNIX's /dev/null). So apparently some APIs will allow you to use those as filenames while others will choke on them.

Thanks, corrected that. Shows what I get for transcribing wrong :)

Yeah, windows is kinda crazy inconsistent for some of these. I had a file (created under Linux) which ended in a space... drove windows nuts. Could list it, open it in some programs, but couldn't even open/rename by shortname under DOS or python.

It is likely that the .NET base class libraries block NUL and the other special file names that Win32 supports. This would explain why PowerShell (which is written in C#) behaves differently from cmd (which is written in C).

Yes the reserved file names based on device names can be a tricky issue for portability. We ran into a problem where a source code file was named Con.java and it was impossible to use that repository on Windows. Had to rename it as Con_.java to make it work.

> Also, ending filenames with space or period really messes up windows. File explorer can see it, but can't delete or rename it.

As a related tip, if you need to name a file something like .foo in explorer, it rejects it as "not having a file name". But if you type .foo. then it accepts the name and silently strips the trailing period.

  `echo missed one`

IIRC, windows has a dialog that shows their full list of disallowed characters if you try to use one of them ... so try to make a file with (eg) "\" in the name and see what the dialog says.

disclaimer: i'm remembering something from the Windows 2003 era, so YMMV.

It's still there, as a balloon popup that says a file name can't contain the following characters:

\/:*?"<>|

As of Windows 7, if you create the file, or folder via raw API calls (Say, from the text editor built into FAR Manager), it's possible to work around this restriction. You can also create folders called "", or " ".

Surprisingly enough, FAR will deal with this 'somewhat' gracefully, but unsurprisingly, Windows Explorer will completely break.

Reply on news.ycombinator.com