Hacker News Reader

Comment by userbinator | Hacker News Reader

I agree that this is a good change. Unicode, normalisation, character encodings, etc. should really be handled at the presentation layer, and everything below that just treats filenames as sequences of bytes, perhaps with one or two exceptions like '/' and \0.

It is interesting to consider a theoretical system in which paths are represented in 0-terminated count-length format (e.g. "foo/bar/baz/myfile.txt" would be "\003foo\003bar\003baz\012myfile.txt\000"), truly allowing any byte in a filesystem node's name, although that might be going a little bit too far.

Things are much easier for the file system if it can just treat names as bags of bytes.

If you're really talking about bags (unordered sets), that would certainly make for an interesting filesystem since filename.txt, filemane.txt, and maletent.fix would all be the same...

View on HN

This feels in some sense like punting the problem. What exactly should the presentation layer do when presenting two files, where the first is named with a precomposed character sequence, and the other has the same name but decomposed? Surface the normalization form the user? Uh, no...

The more fundamental question is whether filenames are under control of the user or the system. The answer today is "both": there's blessed paths /System/Library... and non-blessed paths like ~/Documents/Pokemon.txt. Addressing this properly means reifying that distinction: making apps always be explicit about whether they're working with the user's or the filesystem's view of a file.

While we're here, why the should the user even have to name every file? Do you name every piece of junk mail on your desk/kitchen island?

The ideal for the user is something like tagging, where naming stuff is optional, names are forgiving and perhaps not unique, and files are not restricted to being in one directory. Meanwhile, filesystem names are an implementation detail and the filesystem enjoys Unicode ignorance. Spotlight moved a bit in this direction, but the end goal is still awfully far away.

> What exactly should the presentation layer do when presenting two files, where the first is named with a precomposed character sequence, and the other has the same name but decomposed?

Show two files with apparently identical names. Is that so surprising? There are many many ways for two different Unicode strings to look visually identical (or near-identical) even if they aren't equivalent under normalization. For that matter, even if you were limited to plain keyboard ASCII, you could have two really long filenames that only differ in one character, which are effectively indistinguishable without massive hair-pulling. If you want to be robust, there's no way around having means to identify files other than names.

(I agree with your sentiment about tagging.)

Google drive allows you to have identically named files in the same directory. Doesn't quite map onto a desktop filesystem, but the filename is not a unique identifier, but just a convenience for the user or application.

Even with normalization, you can just sprinkle an arbitrary selection of non breaking zero width space characters into a string and create an infinite variety of identical-looking but distinct codepoint sequences. Treating filenames as bundles of bytes is fine, but at that point it seems dangerous to assume that interpreting them as encoded strings is a valid way of distinguishing them.

You are talking about confusables. Unfortunately normalization cannot handle this. Only certain apps treat confusables as confusables: email, web browsers, dns services, usually via libidn.

> For that matter, even if you were limited to plain keyboard ASCII, you could have two really long filenames that only differ in one character, which are effectively indistinguishable without massive hair-pulling.

My initial response was to not present confusing data to the user, but then I thought of exactly this. In many fonts, lowercase "L" (l) and uppercase "i" (I) look the same. I'm sure in some fonts they actually are identical, rather than just being nearly indistinguishable to the human eye.

To the discussion is not about whether we should present confusing identifiers to users, but at what point if it becomes more common does it cause enough of a problem to be a bad choice, and what techniques are there to mitigate it?

This distinction is important for other reasons, too. Should this file be indexed for search? Should it store an access time to support features like "recently opened"?

On the other side, we want to know which blessed paths "belong" to which system application or service so we can sync them, protect access to them, and uninstall them with their owners.

Maybe the user-visible document store should be a separate mechanism entirely from low-level app/service data persistence.

Microsoft tried to do something like your ideal with WinFS but couldn't make it work and abandoned the project.

I'd argue: Use a langauge with generics and make the path type a ((de-)serializable) black box.

You still need to make a decision in the end, but are much more free to change it.

This is not a language issue.

I didn't say it was. I said generics would help one easily switch their choice on the different semantics, not avoid choosing at all.

Many, or most, languages already do that by representing filenames as native strings (which are usually abstract "Unicode" strings).

I want to swap the path type used by the FS implementation. This is totally distinct from a langauge providing some string type which while abstract is nevertheless fixed and unchangable.

Both Objective-C and Swift have (a form of) generics on board.

It's a good idea until you end up with two files that have the "same" name (eg. Amélie.jpg and Amélie.jpg) because one uses decomposed characters (U+0065 and U+0301) and the other one uses a single character (U+00E9).

If the difference is not visible in your browser (it shouldn't), try copy-pasting those two filenames in a text editor, one of them is 10 characters long and one of them is 11 characters long.

It can get worse - imagine a program that normalizes all paths before handing them off to the fs. Then you go to delete the denormalized file, but it deletes the normalized one. With this change, the only safe way to handle paths would be to NOT normalize at all, but Apple are recommending the exact opposite. I predict a lot of confusion will arise from this change.

The problem exists already on the visual level, since there are pairs of distinct Unicode characters that look very much alike.

I think the only thing you can really do about that is restricting to ASCII minus control characters. Probably many programmers would be willing to accept that, but non-technical users not.

The next best thing would be to enforce a canonical (code-point?) encoding. But given the complexities of Unicode that will get us only so far...

> The next best thing would be to enforce a canonical (code-point?) encoding.

That's exactly what HFS+ did, and what APFS doesn't.

> The problem exists already on the visual level, since there are pairs of distinct Unicode characters that look very much alike.

Yes, but software doesn't consider them equivalent.

The software also doesn't consider the two strings given as an example above equivalent, and for the same exact reason: they're sequences of different code points.

https://turnoff.us/geek/user-space-election/

That's not about filenames.

>It's a good idea until you end up with two files that have the "same" name (eg. Amélie.jpg and Amélie.jpg) because one uses decomposed characters (U+0065 and U+0301) and the other one uses a single character (U+00E9).

It's still good then. And several systems allow for that just fine, including VMS (uniqueness comes from more than the filename). This is more alike real world folders (which can have identical items, e.g. two copies of the same paper), and is also an excellent way to keep different document versions (keep the name the same, change the date shown).

Are you also against case sensitive file systems? Otherwise you can end up with two files that have the "same" name - eg. anne.jpg and Anne.jpg). Does normalization cover such a case?

I think, the problem with Unicode normalization is, input methods are (well, conceptually, at least) meant to produce text, not binary. If filesystems are using binary data for filenames, there can be a case when is really no way to address a file by typing its name, even if you can type in that language. This isn't an issue for case sensitivity or alike.

>If filesystems are using binary data for filenames, there can be a case when is really no way to address a file by typing its name, even if you can type in that language

You could do that trivially in UNIX since forever.

Don't know about other UNIXes, but at least on GNU/Linux, neither IMEs nor filesystems are working with text data - it's all binary strings (with a few restrictions, like unacceptability of NULs). The only place where those binary strings are converted to text is when they're rendered (and this may cause some encoding-related oddities). So, sure one can do that.

I would've actually preferred for identifiers to be Unicode text strings.

Filesystems themselves treating file names and paths at bits of binary data is the right way to do it, normalization shouldn't be handled directly by the filesystem as it adds complexity and duplication of effort.

Ideally, the VFS layer should be handling all this garbage.

Programs often allow you to type arbitrary binary values using a keyboard, either by holding a keyboard control key and typing the numeric value, or prefixing it with '\'.

For ascii, I would gladly do this There may be edge cases outside of the English language that I'm not aware of.

I am somewhat willing to accept the value of case sensitivity in identifiers for programming languages (though it's often related to the verbosity of the language, as in Java: "Foo foo = new Foo()"), but not in file naming.

Of course, I realize that the ship has sailed, and we're stuck with case sensitivity in file systems.

No good has ever come from allowing BEL and DEL as part of filenames.

I'm with David Wheeler: https://www.dwheeler.com/essays/fixing-unix-linux-filenames....

We need to limit filenames for the good of the entire system and the whole community. Filenames as byte strings may sound good, but nobody ever thinks of the costs and the scant benefits.

I disagree completely. What we need is to allow even slashes in the file names – not restrict (for what an end user may feel arbitrarily) even more writeable characters off file names, just so we can avoid fixing our replacing our tools or writing one line more in a shell script.

Any solution you come up with must work with replacing existing software piecemeal, or it's no solution.

> Things are much easier for the file system if it can just treat names as bags of bytes.

And much, much harder for applications if that "bag of bytes" is an invalid UTF-8 sequence. You will end up with an invalid string (or an exception), and trying to open that file will then fail.

I'd really hope that Apple checks that the filenames are valid UTF-8 as they otherwise can end up with very interesting security bugs.

> And much, much harder for applications if that "bag of bytes" is an invalid UTF-8 sequence.

It makes copying-and-pasting filenames as Unicode unreliable, for one thing.

You can handle that fine provided you don't assume that.

> I'd really hope that Apple checks that the filenames are valid UTF-8 as they otherwise can end up with very interesting security bugs.

I think that's a bit pessimistic as Linux has run for year like that without any real issues.

At least valid utf-8 isn't difficult for a filesystem to enforce.

I'm not so sure: the thing is, those strings are not just a bag of bytes. They have semantics. Is it legal to include bytes that are 0? How about slashes in file names? How about byte sequences that aren't utf8 at all?

These things aren't just byte streams, they have semantics.

So one risk of treating this as "just bytes" is that bugs will introduce byte sequences that aren't utf8 at all, which will cause other programs to fail, or worse, to "try" and thereby corrupt the data further.

Another risk is that since it's supposed to be utf-8, some programs may do canonicalization internally to avoid confusing situations. This may even happen accidentally (though it's not likely): after all, a unicode-processing system could be forgiven for transparently changing canonicalization. But if a program canonicalizes you can now get really weird behavior such as opening a file, then saving it, and ending up with two files that look identical - because the write wrote to a path that was canonicalized.

Additionally, even though you can never avoid confusing paths entirely without considering the glyphs rendered, you are losing a very simple check against a fairly large class of errors.

No. Ever heard about http://websec.github.io/unicode-security-guide/

Identifiers should be identifiable. If a filename is encoded in utf8, it needs to be normalized. To the canonical form of course not the crazy Python 3 or Apple idea of NFD. Which is also slower. If it's encoded as bytes you get garbage in - garbage out.

There's much more to consider, but unfortunately you cannot restrict a directory to forbid mixed scripts or confusables. I summarized a few problems at http://perl11.org/blog/unicode-identifiers.html

(Offtopic) You may want to correct that blog post, the point about Japanese writing systems. "[...] Japanese using characters from Chinese (Kanji/Han), Katagana (modern japanase) and Hiregana (the old middle-age script used by women)" is really incorrect. Suggest to just say that modern Japanese uses both logographic (kanji, originated from Chinese hanzi[1]) and syllabic (kana) characters simultaneously, with two distinct types of kana (hiragana and katakana).

[1] In Unicode, they're generally unified into a single set, via process called "Han unification". So, unlike Greek "Α" vs Cyrillic "А", the "same" character that may even look slightly differently in Chinese vs Japanese (e.g. "海"), would have a single codepoint in Unicode. But that's another story...

Ugh… yeah, agreed; reading that was painful.

To reiterate:

• Chinese characters are called hànzì¹

• modern Japanese uses many hànzì, calling them kanji² instead

• katakana³ is used in modern Japanese to write words of foreign origin (loanwords) as well as onomatopoeia and is romanized as katakana, not katagana

• hiragana⁴ is used in modern Japanese to write okurigana⁵, particles, and certain words, and is romanized as hiragana, not hiregana

Here’s a typical (and extremely simple) sentence in modern Japanese that uses all three:

この文はサンプルです。

This is a sample sentence.

この: this (hiragana)

文: sentence (kanji)

は: particle (pronounced wa) (hiragana)

サンプル: sample (loanword; pronounced sanpuru) (katakana)

です: particle (pronounced desu) (hiragana)

――――――

¹ — https://en.wikipedia.org/wiki/Chinese_characters

² — https://en.wikipedia.org/wiki/Kanji

³ — https://en.wikipedia.org/wiki/Katakana

⁴ — https://en.wikipedia.org/wiki/Hiragana

⁵ — https://en.wikipedia.org/wiki/Okurigana

> です: particle (pronounced desu) (hiragana)

(While we're being pedantic)

です isn't a particle, it's the imperfective (present/future) polite form of the copular verb (to be, in English). The only particle in that sentence is は.

And as a further side note, some words or names normally written in kanji or hiragana are ocassionally written in katakana for emphasis or to emphasize a foreign nature.

Thanks, appreciated. I only have two "Learn Japanese" and "Learn Korean in x days" books for dummies.

Fixed

> No. Ever heard about http://websec.github.io/unicode-security-guide/

Which part of it is relevant to pathnames?

> To the canonical form of course

Do you mean NFC?

> not the crazy Python 3 or Apple idea of NFD.

In what circumstances does Python 3 normalize pathnames?

> If it's encoded as bytes you get garbage in - garbage out.

What do you mean by "encoded as bytes"?

> http://perl11.org/blog/unicode-identifiers.html

This is about programming language identifies, not about pathnames, which are very different beasts.

> Which part of it is relevant to pathnames?

All parts regarding identifiers and names, if you see pathnames as publicly identifiable information. See below.

> In what circumstances does Python 3 normalize pathnames?

None. Python 3 normalizes identifiers to the shorter non-canonical NFKC form. Apple HFS+ normalizes pathnames to the longer D canonical form. Which is faster but takes more space. Usually space is more important than CPU.

> What do you mean by "encoded as bytes"?

What everyone else means on byte encodings: 1-1 mapping of bytes, without any further knowledge of the encoding.

> This is about programming language identifies, not about pathnames, which are very different beasts.

Partially. pathnames can also be argued to be identifiers, similar to domain names, email names, user names, language identifiers. Apple does so. Ask plan9 what they think about pathname semantics. Not everyone is in the garbage in - garbage out camp. Many systems do encode pathnames and have character restrictions. Esp. important is the popular / spoof, which people from the garbage camp don't care about. Julia is recent proponent of the garbage camp, btw.

> to the shorter non-canonical NFKC form

What definition or sense of 'canonical' do you mean? NFKC stands for "Compatibility Decomposition, followed by Canonical Composition", so it's canonical in some sense, right?

There can only be one canonical normalization, not two. NFC is the canonical one, NFKC compresses ligatures differently.

> unfortunately you cannot restrict a directory to forbid mixed scripts or confusables

That's fortunate, not unfortunate. I want the files corresponding to documents to be named according to the titles of those documents, and said titles often do mix Latin and Cyrillic scripts in a way that produces "confusables". So as a generic rule for all filenames, it's a no-go - it will significantly affect perfectly legitimate use cases.

For average user "documents" and "Documents" is the same thing. Are paths also normalized to upper or lower case? Or file system should be case insensitive?

Case insensitivity is something best addressed in userspace because:

1) History has shown that doing it at the filesystem level causes significant compatibility issues

2) Case insensitivity can present visually confusing results

3) Round-tripping between cases sometimes is not possible without loss of information

4) Some languages don't have a meaningful definition of 'case insensitive'

5) Some languages would demand a more comprehensive approach (i.e. being able to search using either kana or kanji for Japanese filenames)

So for things like case insensitivity (or #5 above) it's best to handle them at the application (or, ideally, OS user space library) level, taking advantage of information like system locale. Low-level APIs and system services can use case-sensitive filenames.

This is different from the question of whether to normalize filenames, because filenames are still visually unambiguous (at least most of the time) in UI, logs, and debuggers.

Incidentally, Windows is proof of this technique's advantages because NTFS is a case-sensitive file system while Win32 is case-insensitive. IIRC the Linux subsystem for Windows takes advantage of this.

> If you're really talking about bags (unordered sets), that would certainly make for an interesting filesystem since filename.txt, filemane.txt, and maletent.fix would all be the same...

I was thinking the same thing! Although if we're being pedantic, I think "bag" is an "unordered multiset". If it were just a plain unordered set, then "filename.txt" would also be the same as "filenam.txt".

Incidentally, your theoretical system is exactly how the DNS stores domain names.

>perhaps with one or two exceptions like '/' and \0

Aren't colons (:) an issue with MacOS as well? I think Finder and other userspace apps convert them to slashes. I suppose though, there's a case for the filesystem not caring about that.

This sounds like DJB's Netstring [1].

"hello world!" encoded as "12:hello world!,"

An empty string as "0:,"

[1] https://en.wikipedia.org/wiki/Netstring

How do you normalize arab or chinese in a meaningful way for people speaking these languages ?

You don't, there is only one unicode representation of those characters.

Interestingly that created its own annoyances for some since normalization occurs at a different stage.

Unicode and UTF-8 unify many Traditional & Simplified Chinese Characters as well as Japanese Kanji and Korean Hanja.

There were some extremely angry reactions to Han Character Unification[1], though efforts have been made to overcome many of the initial complaints.

[1] "Why Unicode Won’t Work on the Internet: Linguistic, Political, and Technical Limitations" http://www.hastingsresearch.com/net/04-unicode-limitations.s...

Not the case for Arabic. There's hundreds of ligature code points, for example ﰻ (kl), ﷵ (ṣl'm) and ﷳ (akbr). And of course ﷽

I don't believe that's literally true, some all legal and correct Japanese unicode sequences will be transformed under at least some of the unicode normalization forms, no?

So what does mac do with those ?

Use that representation? The problem only happens when there is more than one way to represent the same text.

If I were allowed one more distinction I would definitely add utf-8 only.

The same goes for case-sensitivity which, unfortunately, they see as a defect. It would be great for portability if APFS would stay case-insensitive like the Unix file systems, e.g. just a dumb layer that reads/writes bytes.

I think you mean case-aware, not case-sensitive. Just bytes like Unix leads to case-sensitive behavior.

Reply on news.ycombinator.com