Hacker News Reader

Comment by migueloller | Hacker News Reader

Would you be willing to expand on why using strings for file paths turns out to be unworkable? Very curious what the obscure reasons are!

View on HN

Short version: file paths are an OS-specific construct, and not all OSes have paths that can be round-tripped through Unicode.

The first bad option is to say "paths are bytes", which is true on traditional POSIX platforms. Then you run your code on macOS and discover it's processing your bytes as UTF-8 and applying Unicode normalization. After fixing that you try porting to Windows and get to learn about `wchar_t`.

The other bad option is to say "paths are (Unicode) strings", which works fine on Windows and macOS. It works 90% of the time on POSIX, but you'll eventually find users with non-UTF8 file paths and then you're in trouble.

My first hand experience with this going wrong is GHC, which started off with raw bytes, then moved to strings using a bad conversion function, then they tried to work around the issue by placing non-UTF8 bytes into a Unicode reserved area.

Java also has problems because it represents paths as strings and tries to use the libc locale for decoding. https://github.com/bazelbuild/bazel/pull/10111 is an example of how this breaks software in ways that are difficult to work around.

Thanks for the response!

If your String type must be valid Unicode (which is true for pretty much every language that isn't C++), it cannot represent all paths. On most unixy filesystems, filenames are arbitrary byte sequences which aren't valid UTF-8, and on windows, NTFS stores filenames as UTF-16, but it allows unpaired surrogates.

Is that technically UCS-2 rather than UTF-16?

Technically not since ~2000. Windows uses UTF-16 throughout the OS. That is, Windows strings will be encoded and decoded as UTF-16 not UCS-2.

However the tricky bit is that the kernel doesn't enforce this so it's possible for programmers to intentionally make broken UTF-16 strings. A broken UTF-16 string shouldn't be considered UCS-2 just because it happens to be a valid UCS-2 string (otherwise all bit patterns could be called "UCS-2" so long as they are an even number of bytes in length).

What languages require string types to be valid Unicode? Go for example let’s you put anything in a string; it’s only Unicode if you put Unicode into the string including via creating string literals.

Rust, Java, Swift...

I think python also takes the approach where strings must be valid unicode. Both Rust and Python use their String types to represent valid unicode, and something else ([] and byte strings, respectively) to represent other encodings and invalid unicode.

Requiring strings to be valid unicode simplifies the internal workings of (and use of) a language greatly. Dealing with unknown input becomes a single operation at the boundary after which you can safely assume everything is validly printable or processable as text.

I think there is some convenience in being able to use the type system to denote input which has already been validated; however, I can't see how a language runtime benefits from this property with respect to unicode. Notable, printing and processing text aren't operations a runtime typically deals with. Libraries including the standard library might benefit from this property, but even then I wouldn't say the benefit is "great". It's a nice property, but I don't think it makes or breaks anything.

It's worth pointing out that Python 3 thought it important enough to make a backwards incompatible change to separate 'bytes' from 'strings' to effectively get the same result.

Probably, but I think that’s more to do with the typing convenience (which is even more puzzling in Python’s case since it isn’t statically typed).

(To add on to the sibling comment's great answer, here's Rust's docs on this https://doc.rust-lang.org/stable/std/path/index.html and https://doc.rust-lang.org/stable/std/ffi/struct.OsString.htm... )

More interesting is what you do use for file paths, if you accept that both strings and bytes don't work. The answer will be I guess a dedicated FilePath type that knows paths are lists of blobs, and the blobs can further be divided up into prefixes and an extensions.

But that doesn't really solve the problem. You still have to convert file paths to strings to print them, and strings to file paths to accept them as from the terminal and databases. For that to work the round trip had better be safe on any given platform otherwise the user can't copy and paste.

Not addressing this problem has been python3's undoing when it is used as a shell scripting replacement. Actually, it's an undoing any time you want to treat "text" as interesting ascii swimming in an unknown encoding, particularly when you want to modify that ascii. HTML with an unknown encoding is a prime example.

Reply on news.ycombinator.com