Hacker News Reader

Comment by sakras | Hacker News Reader

I’m a little confused, how can a file name be non-decodable? A file with that name exists, so someone somewhere knows how to decode it. Why wouldn’t Python just always use the same encoding as the OS it’s running on? Is this some locale-related thing?

View on HN

> A file with that name exists, so someone somewhere knows how to decode it.

No. A unix filename is just a bunch of bytes (two of them being off-limits). There is no requirement that it be in any encoding.

You can always use a fallback encoding (an iso-8859) to get something out of the garbage, but it's just that, garbage.

Windows has a similar issue, NTFS paths are sequences of UCS2 code units, but there's no guarantee that they form any sort of valid UTF-16 string, you can find random lone surrogates for instance.

And I'm sure network filesystems have invented their own even worse issues, because being awful is what they do.

> Why wouldn’t Python just always use the same encoding as the OS it’s running on?

1. because OS don't really have encodings, Python has a function to try and retrieve FS encoding[0] but per the above there's no requirement that it is correct for any file, let alone the one you actually want to open (hell technically speaking it's not even a property of the FS)

2. because OS lie and user configurations are garbage, you can't even trust the user's locale to be configured properly for reading files (an other mistake Python 3 made, incidentally)

3. because the user may not even have created the file, it might come from a broken archive, or some random download from someone having fun with filenames, or from fetching crap from an FTP or network share

There are a few FS / FS configurations which are reliable, in that case they either error or pre-mangle the files on intake.

IIRC ZFS can be configured to only accept valid UTF-8 filenames, HFS(+) requires valid unicode (stored as UTF-16) and APFS does as well (stored as UTF-8).

[0] https://docs.python.org/3/library/sys.html#sys.getfilesystem...

It's filesystem-dependent, but a lot of filesystems treat filenames as just an arbitrary sequence of bytes. Most OSes sort of hide this from you, but there's nothing stopping you doing weird stuff like:

  > touch `echo 00: DEADBEEF | xxd -r`
  > ls
  'ޭ'$'\276\357'

Unix allows any byte in a filename except 0x0 (ASCII Nul) and 0x2f (ASCII '/'), anything else is allowed. It doesn't have to be decodable to text.

Unfortunately, neither UNIX nor Windows require path names to be valid Unicode. UNIX interprets them as “UTF-8 by convention” and Windows as “UTF-16 by convention” but both actually allow arbitrary sequences of code units. It would be nice if this didn’t actually occur, but alas, it does, and if you’re writing general purpose utilities that work with files, you don’t want them to simply crash when this happens.

Does trying to decode, on Unix, a Windows file path (not filename) like "drive:blah" freak it out as an example of non-decodable?

That's a valid file name. Non-decodable are byte sequences which are invalid UTF-8 byte sequences (if UTF-8 is expected)

Reply on news.ycombinator.com