Hacker News Reader

Comment by jefftk | Hacker News Reader

APIs don't do well with "probably". For example, say you're working with a language (Python, Rust, etc) that distinguishes between utf-8 strings and byte strings. You're making an API for listing a directory: it gives you an array of strings but what kind should they be?

(Python's approach is that os.listdir("/") gives you a List[str] (silently omitting undecodable entries), while os.listdir(b"/") gives you a List[bytes]. That is, if you give the path as a utf-8 string it returns utf-8 strings, otherwise it returns bytes.)

View on HN

If it's an API for "listing a directory" the things in it aren't strings they're paths, and Rust indeed gives you Paths here (actually PathBufs in case you want to do stuff to them)

Paths might just be strings, but they aren't necessarily, and since Rust actually cares about types if you want strings you need to write the code to decide what to do about this, even if your "It's not UTF-8" case is just "Give up I can't be bothered".

Yes, Rust in keeping with its aesthetic resolves the "probably" in the most pessimistic direction, and requires you to explicitly say how you want to handle the potential for non-utf8.

Nitpick: Python strings are not "UTF-8"; they are abstract sequences of Unicode codepoints (and internally CPython stores UTF-32). However, UTF-8 is the default encoding for processing raw bytes received from the outside world and turning them into strings.

That said, I actually didn't realize that os.listdir() silently omits un-decodable entries, which I find mildly alarming. This behavior isn't mentioned in the docs (https://docs.python.org/3/library/os.html#os.listdir) and seems out-of-character for Python, which usually raises an exception by default if data cannot be decoded to text.

Are you sure that this is actually what happens with non-decodable filenames? Reading here (https://docs.python.org/3/glossary.html#term-filesystem-enco...) and here (https://docs.python.org/3/c-api/init_config.html#c.PyConfig....), it suggests that encoding errors should be handled by surrogate escapes by default on non-Windows systems.

I'm wrong: https://news.ycombinator.com/item?id=32471944

If it ever did that, it doesn't anymore.

Found some history: https://vstinner.github.io/python30-listdir-undecodable-file...

Includes "Modify os.listdir(str) to ignore silently undecodable filenames, instead of returning them as bytes", but not later work where apparently this was changed to use surrogates.

Just as a note, the way that Python stores the items isn't always 4 bytes per char, it depends on the actual string contents.

I think that https://rushter.com/blog/python-strings-and-memory/ is a nice reference on that.

Interesting and good to know! CPython is really quite impressive with all of these little optimizations.

Honestly? Convince operating systems to have a switch that enforces UTF-8 path names, and then convince distros to flip that switch by default for new installs. That is to say, we need to move the world from a probably to a definitely state.

The reality is that file names are "stringy" in nature--people expect to be able to do display them--and that means you need to have some up-front agreement on how to interpret those strings. In practice, on Unix systems, everyone has generally agreed that this is UTF-8, to the point that trying to not be UTF-8 generally causes interesting breaks in the system. It would be great if we could actually get the operating system to help enforce these rules, rather than placing the blame on other software for not correctly handling situations where the correct solution is itself incredibly ambiguous.

While I am all for an OS monoculture, it's still not there yet... and besides, my idea of what OS precisely must be the sole survivor is different from yours, and yours is different from somebody else's, etc.

So I am afraid we can either a) indulge ourselves in wishful thinking, b) actively try to extinguish platforms that don't match our ideals, c) make an effort to be actually cross-platform, and not in "let's just build a tiny Linux model in a bottle for us to use and pretend the rest of the environment is not there" kind.

On a system with that switch flipped, what should happen if you plug in a USB drive or untar an archive that has non-utf8 filenames?

For a legacy/broken FS that permits non-utf8 filenames, and has them: have the FS driver map them into UTF-8 as best it can by making some sort of compromise. E.g., use the PUA to map malformed sequences in/out.

For untar'ing a tar archive: error out by default, but provide a flag or option to permit untarring using some sort of escaping to map the malformed names back into Unicode. I think here I'd map to something printable, though, like "\xnn" or something.

You can use ZFS as a root file system, and it actually has such a switch (called "utf8only").

They should be byte (or 16-bit code unit, or whatever) strings. There is no ambiguity here, only incompatibility and delusion.

Rust gets this almost right. Python gets this very wrong.

> Rust gets this almost right. Python gets this very wrong.

Having worked in both, I'd say they both chose ideomatic solutions:

Rust: I can't prove this is utf-8, so if you want to use it as utf-8 you'll need to tell me what to do if it isn't.

Python: if you're in the common situation where everything is utf-8 and you want to just work with strings, go do the simple thing. Or you can be explicit about wanting to work with bytes, and that's good too.

(Though I think Python should throw an error instead of silently omitting non-utf8 files.)

Actually, I think I'm wrong about python's behavior, at least now; it doesn't omit the files, and instead does something with surrogates. On Linux:

    >>> b'\xc0'.decode('utf-8')
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
    UnicodeDecodeError: 'utf-8' codec can't decode byte 0xc0 in position 0: invalid start byte
    >>> open(b'\xc0', 'w').write('foo')
    3
    >>> os.listdir()
    ['\udcc0']
    >>> open(b'\xc0').read()
    'foo'
    >>> open('\udcc0').read()
    'foo'

PEP 383[1] is a workaround, but it took a while to get there[2].

[1]: https://peps.python.org/pep-0383/ [2]: cf. https://github.com/bup/bup/blob/master/DESIGN#L667-L729

I just did a bit of research into this here https://news.ycombinator.com/item?id=32472087

You can actually reconfigure Python to throw an error instead of using a surrogate escape, but only (I think) changing something at compile time: https://docs.python.org/3/library/sys.html#sys.getfilesystem...

Reply on news.ycombinator.com