The only sane way to deal with them all is to use bytes and bytes only (Or use a sane language with encoding-agnostic string type; see: golang) Other solutions introduce loss of data.
It's not really, though. Yes, if you want to round-trip the name of the file and be fully general over the bad situation provided by the OS. But as soon as you want to display that filename, or really do much of anything with it, you're back into the world of hurt the article describes, and you need some super complicated type or logic that can convey to the user "hey, this file's name is some random undecodable garbage".
The sane state would have been "file names are text", but filenames/Unix predates software engineering having a good discipline around text encoding, and it predates Unicode. I'd go as far as to argue that "file names are text and \n is disallowed" is the sane state, as multi-line filenames also just cause all sorts of unnecessary complexity.
The problem, of course, is that most OS enforce neither constraint, which is what leads to things like the struggle Python has gone through, or the Path type in Rust, and the unrealistic nature of trying to move from the present state to such a sane state. (Linux is loathe to break compatibility, and I bet Windows would be similar, here. There are a number of arguments for that, but in the meantime, it continues to be that file names are ridiculously complicated if handled fully correctly.
Or, you just decode them as Unicode (UTF-8/-16 depending on OS) and give up if the user is crazy. 99.99% of the time that'll work just fine.)
> Or use a sane language with encoding-agnostic string type; see: golang
Of course, this might be the root of our disagreement. Types that permit the representation of invalid values are not desirable, to me. Especially in a core type, like strings.
You have to be really careful about the way in which you give up here, as the user isn't necessarily in control of the files on disk: it might be an attacker. If I can hide files in folders from various parts of a Python program -- as was possible with early versions of Python 3, which simply skipped such files -- or cause corruption in how the filenames are round-tripped during a copy (allowing me to bypass a filter in one place but then write to a file with a different critical name in another place) I gain a lot of flexibility with my exploits.
(But, that might also fall into the 1% case where you'd need to pay attention.)
Perhaps you might argue for a dedicated type like Rust's Path, but that too has issues, and I don't think Python had the "design freedom" to go there. (Although it does have pathlib, nowadays, so I wonder how it handles this; but it would also need to duplicate a lot of the functionality in os…)
IMHO, that is mostly not the file system's problem. Bits going into and out of the file system should act as deterministically as possible.
If you want to start being 'clever' about these problems, do it in userland.
Yeah, an FS could say that¹. Alternatively, it could say that names are intended to be read by humans, and that names should be a single line, to support things that people reasonable expect to be able to do with a file system, like being able to list the files on the filesystem, without whatever is doing the listing needing to decide if it is going to just say "I can't display raw bytes, obviously, so here's a best-effort of ���.txt" or if it has to invent some sort of grammar to encode/escape bytes into text.
I.e., a file system is meant to solve a problem, which is the storage and organization of data, and from that, that names are meant to used by humans is one of the requirements of both the OS & the filesystem. (The OS, as it presents the file hierarchy to userspace, e.g., in the VFS, and the filesystem, as it of course needs to store the data.)
By your logic, we should permit nul and '/' in file names, too.
¹and, I mean, most do. Which is why we're in the situation we're in, and why languages are faced with a barrel of bad choices, and why probably 90% of shell scripts fail to properly handle file names. Is this what we want?
If you want to fix shell scripts, you need to ban not just newlines but also spaces. Well-written shell scripts handle spaces properly, but many don't, and Makefiles cannot.
It really is. Once you are using byte type you can handle displaying byte type to user as you see fit but that's not a decision stdlib should make for you. Not only that, if any file-related function operates on bytes that by definition it doesn't care about FS semantics, you're also not wasting any CPU for useless conversions to string.
The blob of bytes acts as opaque token for file location, as it should. If you want to display it to users you have to quote it anyway, because you can have perfectly valid utf8 file name that contains newline or some special utf8 characters that will fuck up how it is displayed and might even in malicious way
>> Or use a sane language with encoding-agnostic string type; see: golang
>Of course, this might be the root of our disagreement. Types that permit the representation of invalid values are not desirable, to me. Especially in a core type, like strings.
string is defined as "read-only slice of bytes", not "UTF8 slice of bytes". That is mostly so conversion between is free. If you want to pay the cost of validation you can, but are not forced to.
The functions operating on them, if relevant to the function's functionality, will operate in UTF8.
It might sound bad theory and it did to me at first but in practice it's probably the least problems I had out of all languages.
Something fun on file systems: ZFS lets you set "utf8only=on" which will guarantee no file names will ever be possible that are not valid UTF-8. I use it on all my systems.
This is the sort of option that should be available for all filesystems, and distros should default new installs to turning it on.
But alas, we do not and someone will plug in a memory card or USB drive with some FAT formatted drive and your nice universe crumbles before you.
The lesson of text is clear: text needs to have a defined charset. Filesystem names are absolutely textual (listing a directory for human display is one of the most common operations). If it's not UTF-8, then what is it? How is the application supposed to figure it out? (Automatic encoding detection is flaky in the best of times, but with the lengths of typical filenames, it is almost impossible to be reliable). The reality is that most applications already assume that filesystem names are UTF-8, and non-UTF-8 names break them.
We are at a stage in Unicode penetration that I think it is reasonable to treat a filesystem that has non-UTF-8 filenames as a problem that needs to be solved with some kind of repair tool like fsck, not something that every application is supposed to worry about.
PS: unless your take is to say that we should just assume everything is ascii and don't ever render codepoints > 127?
The other candidate is the atrocious operator precedence. OK, we now love parens to 'fix' this -- might as well use lisp... ;-)
The Mesa guys at PARC did a study back in the late 70s and, statistically, length-marked strings are faster than delimited ones.
Delimited strings have some advantages: they save the space needed to store the length, at the cost of an extra byte (not worth the cost these days); and some substrings can be assembled simply by pointing into the string (of course counted strings can share storage). But at a terrible cost.
But remember that the PDP-7 was less powerful than some AVRs. We use more powerful chips to scan keyboards these days.