Hacker News Reader

Comment by jcranmer | Hacker News Reader

> Something fun on file systems: ZFS lets you set "utf8only=on" which will guarantee no file names will ever be possible that are not valid UTF-8. I use it on all my systems.

This is the sort of option that should be available for all filesystems, and distros should default new installs to turning it on.

View on HN

Please don't unless you've thoroughly tested in multiple cultures. Unicode is broken for Japanese; on some systems it's possible to change a setting to make it broken for Chinese instead, but the only non-theoretical way to make a program that can list a directory full of Japanese filenames and a directory full of Chinese filenames correctly is to have that program be encoding-aware.

I'm curious, what does 'broken' mean? Is this about the CJK merger, or something else?

Yes, CJK unification. Characters will be rendered differently from how they should be (e.g. with missing or extra strokes).

Why is that a filesystem issue or a "utf8only=on" issue? If the renderer doesn't have sufficient context, it will pick one glyph to render anyway, and it could be "wrong".

PS: unless your take is to say that we should just assume everything is ascii and don't ever render codepoints > 127?

If the filesystem can't allow parts to be in different encodings then it can't ever possibly work right. With current filesystems at least I can do "ls ~/chinese-files; LANG=ja_JP.SJIS ls ~/japanese-files". Which is not a great way of working, but it's better than a utf8only filesystem where it's physically impossible to make even that much work.

If you run a ZFS file system where this is a huge deal to you, then don't set utf8only. If it's set on a parent dataset, then explicitly do utf8only=off.

if we have a time machine then yes. In fact, it shouldn't even be an option. UTF-8 period. Always.

But alas, we do not and someone will plug in a memory card or USB drive with some FAT formatted drive and your nice universe crumbles before you.

Handling non-UTF-8 pathnames via a filesystem mount option is completely viable. Hell, I'd go even further and suggest that the kernel itself reject non-UTF-8 pathnames entirely, handling filesystems with the non-UTF-8-pathname option enabled via a translation layer that converts \x00-\xff to \u0000-\u00ff (and vice versa).

The lesson of text is clear: text needs to have a defined charset. Filesystem names are absolutely textual (listing a directory for human display is one of the most common operations). If it's not UTF-8, then what is it? How is the application supposed to figure it out? (Automatic encoding detection is flaky in the best of times, but with the lengths of typical filenames, it is almost impossible to be reliable). The reality is that most applications already assume that filesystem names are UTF-8, and non-UTF-8 names break them.

We are at a stage in Unicode penetration that I think it is reasonable to treat a filesystem that has non-UTF-8 filenames as a problem that needs to be solved with some kind of repair tool like fsck, not something that every application is supposed to worry about.

This is why "convmv" exists :) and also why my /tmp is a normal tmpfs where file names can be any byte sequence. I can, for instance, extract random archives there, use convmv to make them UTF-8 (if not already), and then it can go on my ZFS file systems.

Reply on news.ycombinator.com