Hacker News Reader

Comment by jmgao | Hacker News Reader

If your String type must be valid Unicode (which is true for pretty much every language that isn't C++), it cannot represent all paths. On most unixy filesystems, filenames are arbitrary byte sequences which aren't valid UTF-8, and on windows, NTFS stores filenames as UTF-16, but it allows unpaired surrogates.

View on HN

What languages require string types to be valid Unicode? Go for example let’s you put anything in a string; it’s only Unicode if you put Unicode into the string including via creating string literals.

Rust, Java, Swift...

I think python also takes the approach where strings must be valid unicode. Both Rust and Python use their String types to represent valid unicode, and something else ([] and byte strings, respectively) to represent other encodings and invalid unicode.

Requiring strings to be valid unicode simplifies the internal workings of (and use of) a language greatly. Dealing with unknown input becomes a single operation at the boundary after which you can safely assume everything is validly printable or processable as text.

I think there is some convenience in being able to use the type system to denote input which has already been validated; however, I can't see how a language runtime benefits from this property with respect to unicode. Notable, printing and processing text aren't operations a runtime typically deals with. Libraries including the standard library might benefit from this property, but even then I wouldn't say the benefit is "great". It's a nice property, but I don't think it makes or breaks anything.

It's worth pointing out that Python 3 thought it important enough to make a backwards incompatible change to separate 'bytes' from 'strings' to effectively get the same result.

Probably, but I think that’s more to do with the typing convenience (which is even more puzzling in Python’s case since it isn’t statically typed).

Is that technically UCS-2 rather than UTF-16?

Technically not since ~2000. Windows uses UTF-16 throughout the OS. That is, Windows strings will be encoded and decoded as UTF-16 not UCS-2.

However the tricky bit is that the kernel doesn't enforce this so it's possible for programmers to intentionally make broken UTF-16 strings. A broken UTF-16 string shouldn't be considered UCS-2 just because it happens to be a valid UCS-2 string (otherwise all bit patterns could be called "UCS-2" so long as they are an even number of bytes in length).

Reply on news.ycombinator.com