Hacker News Reader

Comment by re | Hacker News Reader

Further reading:

* https://hoytech.github.io/truncate-presentation/ / https://metacpan.org/pod/Unicode::Truncate

* https://unicode.org/reports/tr29/#Grapheme_Cluster_Boundarie...

Truncating at codepoint boundaries at least avoids generating invalid (non-UTF-8) strings, but can still result in confusing or incorrect displays for human readers, so for best results the truncation algorithm should take extended grapheme clusters into account, which are probably the closest thing that Unicode has to what most people think of as "characters".

View on HN

To avoid this and a bunch of other confusion, when accepting user input I recommend normalizing it to the composed form before writing to a DB or file. While unicode-aware tools and software should handle either form just fine, you probably want to avoid that there's something in the pipeline somewhere that treats the decomposed and composed form of the same string as different.

It's good advice to normalise to pre-composed form, but that doesn't solve the problem the previous poster mentioned as not everything exists as a composed form. That said: most things do have a composed form, so you can probably get away with it – right up to when you can't.

Yeah, working an a library system our path was to compose everything (taking into account of course that the octet sizes specified in the directory may or may not actually be accurate depending on whatever system produced the record) and around the same time deprecate any pretense we had of supporting MARC-8.

How does that affect filenames?

IIRC the lower levels of Windows will happily work with filenames that are not valid Unicode strings, for example if you use the kernel API rather than Win32.

But what about Win32? If you create a file before normalization and then open it using the normalized form, will it open the same file or return file not found?

What about other systems? For example AWS' S3 allows UTF-8 keys, with no mention of normalization[1].

On the phone so can't try myself right now.

Anyway for general text I agree, but for identifiers, filenames and such I prefer to treat them as opaquely as possible.

[1]: https://docs.aws.amazon.com/AmazonS3/latest/userguide/object...

No need to go to the kernel API to create filenames that are invalid UTF-16. The Win32 API will happily let you do it.

I was AFK so couldn't check, but yeah you're right.

Just made two files named ä.txt which happily sat next to each other, one being NFC and other NFD.

So yeah, don't mess with the normalization of filenames.

They're both valid UTF-16, though. Can you create a filename with only half of a surrogate pair in it?

I don't use Windows, so I can't check. Linux literally allows any arbitrary byte except for 0x00 and 0x2F ('/' in ASCII/UTF-8). It's a problem for programming languages that want to only use valid Unicode strings, like Python. Rust has a separate type "OsString" to handle that, with either lossy conversion to "String" or a conversion method that can fail. Python uses the custom use Unicode range to represent invalid byte sequences in filenames. It's all a mess. JavaScript doesn't give a damn about the validity of their UTF-16 strings.

(Note that Rust's OsString is different from it's CString type. Well, I guess under Unix they're the same, but under Windows OsString is UTF-16 (or "WTF-16", because it isn't actually valid UTF-16 in all cases).)

Yeah seems to be same with Win32.

I tried using U+13161 EGYPTIAN HIEROGLYPH G029[1], which resulted in a string of length 2 as expected.

Using both chars (code units) and just the first char (code unit) worked equally fine. In Windows Explorer the first one shows the stork as expected, while the second shows that "invalid character" rectangle.

So yeah, treating filenames as nearly-opaque byte sequences is probably the best approach.

[1]: https://en.wiktionary.org/wiki/%F0%93%85%A1

But ä.txt is a perfectly normal filename! WTF is there to "normalize" about that?!?

Äh, nu går vi och gör något annat.

Either ä as a single code unit or a + ¨ as two code units.

Win32/NTFS treats them as two different filenames, as it doesn't normalize them before storing/comparing.

> What about other systems?

The Linux kernel doesn't validate filenames in any way, so a filename in Linux can contain any byte except 0x2F ('/', which is interpreted as directory separator) and 0x00 (which signals the end of the byte string).

ETA: of course some file systems have other limitations, for example '\' is not valid in FAT32.

Not all grapheme clusters have composed forms so normalization doesn't actually gain you anything here.

> Not all grapheme clusters have composed forms so normalization doesn't actually gain you anything here.

Just because the worst case can't improve doesn't mean that making the average case better is worthless.

I saw something about Arabic text, where that naive truncation at codepoint boundaries turns one word into a different word! Like the sequence of codepoints generate something that is represented as a single glyph in fonts, but truncated its totally different glyphs. I don't remember more details, I don't know any Arabic, but grapheme clusters aren't just about adding diacritics to latin characters. In other languages it all might work quite differently. So truncating at word boundaries (at breakable white-space or punctuation) is probably best. Though of course that way you might truncate the string by a lot. shrug-emoji

(I don't think the talk where the stuff about Arabic was mentioned was Plain Text by Dylan Beattie, but I haven't re-watched it to confirm. So maybe it is. Can't remember the name of any other talk about the subject right now.)

Randomly truncating words can have the same effect in any language. It's outright trivial to find examples in English or German. I don't understand why one has to invoke Arab script for a good example.

Yes, but you don’t end up with different glyphs. Arabic script has letter shaping, that means a letter can have up to 4 shapes based on its position within the word. If you chop off the last letter, the previous one which used to have a “middle” position shape suddenly changes into “terminal” position shape.

I'm thinking even bog-standard European umlauts, cedillas, etc go multi-byte in Unicode? (Take a string of ÅÄÖåäöÜü and chop it off at various byte limits and see.)

I don't remember the details, only that it was a bigger deal than with umlauts. I'll see if I can find the talk again.

This is just the general behavior of truncating strings by code point when they contain decomposed glyphs. This can also impact accents etc.

The emoji "" can't be normalized further -- it's a "" followed by a "". If you just split on code points rather than grapheme clusters, even after normalizing, your naïve truncation algorithm will have accidentally changed the skin colors of emoji. Or turned the flag of Norway into an "". Or turned the rainbow flag into a white flag .

EDIT: oh lord Hacker News strips emoji. You get the idea even though HN ruined the illustrations. Not my fault HN is broken.

Presumably referring to Fitzpatrick modifiers.

HN is not broken, it's working as designed.

[poop]

It makes technical conversations about Unicode ridiculously annoying. It's working as designed and the design is broken.

Whether the absence of emojis on HN is a feature or a bug is arguable. But if you can't figure out a way to work around this constraint (e.g. put your example literally anywhere else on the web and post a link here) HN is probably not a good fit for you.

Reading a discussion thread where each message is just a link to some pastebin with the actual message isn't very nice. Besides, I wasn't going to write the message again after HN removed arbitrary parts of it, hence the edit; I think people got the gist. You may feel that discussion about Unicode doesn't belong on HN but I feel otherwise.

Reading discussion threads full of silly emojis isn't "very nice" either, at least for a certain kind of audience. It's a tradeoff, and the powers that be at HN have decided to optimize for sober discussion over expressivity. It's a defensible decision. Keeping HN from degenerating into Reddit is already hard enough.

I'm sure you'd have survived my unsensored message.

Of course. It's everyone else's emojis that would get annoying.

More sites should have this bug. Whenever I see a colored icon in the middle of black text I get inexplicably angry.

Might I suggest therapy?

Even if you only support scripts for which Unicode has composed codepoints, these days you likely can’t get away without properly handling emoji, and there are no precomposed versions of all the numerous emojis that are made of multiple code points (eg. skin color and gender variants as well as flags).

For actual best results you’d probably want to truncate at the word or syllable boundary, and it should likely be language specific.

Reply on news.ycombinator.com