Hacker News Reader

> It's a problem with Unix filenames where the encoding is just a convention

UNIX filenames are NOT necessarily printable text, they're byte strings. Don't treat them as printable text. They're sequences of bytes not containing 0x00 or 0x2F, but with no encoding.

Same for Windows registry keys. Don't mistake byte strings for text.

Text is a byte string with an encoding which describes which byte values are valid and which characters each byte (or sequence of bytes) corresponds to. If the encoding information is discarded, it stops being text and becomes just a byte string.

> UNIX filenames are NOT necessarily printable text, they're byte strings. Don't treat them as printable text.

But they nearly always also happen to be printable text. Treating them as printable text is soooo convenient 99.9% of the time.

Having them be almost printable text, but not quite, is an API design wart that is begging to be used incorrectly. It's the inverse of an affordance.

Sure, I can agree with that. Just because the API is bad doesn't mean you can treat it as though it's the API you wish it were.

OTOH, if "stop whining about how hard the system is to use correctly, just pay attention, be careful, and don't write bugs" worked, we wouldn't have issues like the one this thread is about, and no-one would have needed to invent Rust.

Oh, 100%. It's critical to wrap the crappy APIs into good ones. But that means you have to write the wrapper, and do so very carefully. Then hope you can get your wrapper adopted widely enough to be standardized, and have the old crappy API deprecated & removed (unlikely). In practice, the crappy API will be kept for backwards compatibility, and will forever remain a footgun. You have to be aware of it, even if only in a "here be dragons" manner.

The POSIX filename rules are a "here be dragons" sign. If you can avoid dealing with them, it's best to do so. If you can't avoid it, you'll need to parse your inputs into valid text, and fall back to a byte string handling path (or just fail with error) if they're not text. You can't safely just assume they're text & treat them as such, it doesn't end well.

To be honest, I disagree. Unix filenames are printable text, they're just text that the OS chooses not to enforce any validation on. Especially since the charset is implied by your choice of locale, which is a user decision, not a kernel decision. But we've transitioned into a world where pretty much all systems have settled on UTF-8 as the charset of choice, and the OS's refusal to even permit kernel options to forcibly validate filenames as UTF-8 is starting to look like a poor decision.

(IIRC, at least one of the BSDs has actually moved to forcing filenames to be UTF-8 and refusing path names that aren't UTF-8. Would only that Linux moved down that path as well so that we could be done with this farce.)

A filename consisting of nothing but ASCII Bell characters (0x07) is valid. Those are non-printable characters that (used to) make a sound from the PC's speaker. POSIX filenames can be sound, not text.

I'd agree it'd be nice if we could restrict filenames to valid UTF-8. But that's not the API that existing filesystems provide, nor what (most of) the existing OSes enforce.

Note that BEL(0x07) is valid UTF-8.

Kernel developers need to consider backwards compatibility. You won't want to see some users lose their data because they upgraded the kernel. Therefore it is very hard to "force" something.

Filesystem mount option, have utilities default it to on when creating new partitions.

That's an important way of looking at it, and is correct as far as traditional UNIX operations like fopen go. But it isn't the whole story because many other operations require treating paths as text. For example, converting them to URIs, putting them in ZIP files, or looking up a file on a filesystem which internally stores filenames in UCS-2.

Threading both of these needles at once basically requires viewing paths as potentially-invalid encoded text.

You can tell .decode what to do with errors so it won't throw an exception, the default is 'strict'. I think python 3.6+ did a pretty good job with it overall.

Don't get me wrong it's still painful and annoying and bug prone, but the point is, it's encodings, it's always going to suck no matter what.

>using .decode requires that your input has a valid, known encoding which is rarely true in the real world of messy data.

How else would you decode a string without knowing its encoding? You can either guess (and risk invalid result/decode errors) or store this information somewhere. This is universal and true in every language. In most cases today people choose to guess utf-8.