Hacker News Reader

Comment by kevin_thibedeau | Hacker News Reader

Unicode support requires incorporating their database into a library. At a minimum you need to know which code points are combining chars. For a language with five to ten year update cycles should everyone be stuck with outdated data if the Unicode standard is revised in the interim?

View on HN

All operating systems have the unicode database saved somewhere. There should just be a standard way of accessing it. Just like filesystem.

Edit; that is; it does not have to be linked in the standard lib. Can be a data file somewhere, or a a shared lib.

There's precedent for this, too: time zones! Time zone data can change over time, and as such it's typically stored in system files and loaded at runtime, rather than being embedded in executables.

Locales have some similar behavior as well.

Like the timezone database, many things of which ship(ped) with their own copy.

It's funny how much time I (and probably a number of others) spend undoing this decision and making things all refer to one copy of the data. There's usually also one per programming language/runtime, so the list only grows as new languages/runtime keep coming into the picture.

Everyone's so good at patching vulnerabilities, but did you plan for your app breaking in production for all users in particular timezones because a random service in a chain of a 6, say, re-implemented in Go last quarter, or maybe... changed their Node app to port from using Moment to the standard Intl object 10 months ago, thus depending on the ICU C++ library, and a separate copy of the data that is no longer correct? Try to assess that risk and your head will hurt. These are the sharp edges in a polyglot environment that usually aren't considered when deciding to bring in new languages.

Unicode CLDR has the same issue, but the pain usually isn't acute because it takes much, much longer for new glyphs or other data to appear in common use or start appearing in data feeds or from OS APIs/input. As long as you're at least on some update schedule, it's usually fine, and there's usually fewer things to update.

Yeah, updates to the Unicode database are generally pretty low-impact. For some time now, the properties of all existing characters have been frozen -- the updates are just adding new characters, most of which are either 1) extremely obscure languages that you'll probably never run into or 2) emoji. Either way, you're unlikely to see anything go horribly wrong because you're using an older version of the Unicode standard.

Time zones, on the other hand...

Have Unicode character properties ever not been frozen? If I remember correctly, there are some misnamed characters from 1.0 that still remain as such in perpetuity for the sake of compatibility.

Conversion between encodings doesn’t require a database or knowledge of combining characters.

This. But it strenghtens the arguments that programming environments should just come with some sort of support for the most common encoding forms.

It does: say, Latin-1 has one-byte code points for characters like ê, but a UTF-8 sequence may represent it as an e followed by a combining circumflex.

Latin-1 is a different character set; you do not need knowledge of code points to convert between different UTF encodings.

That does not require you to know which Unicode characters are classified as combining marks. It only requires that a Unicode sequence U+0045 U+0302 is translated to Latin-1 character 0xEA (or vice versa).

A library that only extracts code points will do more damage than not having one at all. If you have to decode Unicode you presumably want to parse it some of the time. Not supporting the needs for string processing with multi-point graphemes leads to broken Unicode "support" that doesn't actually work with all valid Unicode.

Plenty of applications just need to take strings in one format and pass them along in another format without doing feats of string processing to them.

To elaborate: UTF-8 is the most common I/O encoding, but UTF-16 is often what you need to work with the OS APIs (Win32, macOS) or popular frameworks (Qt).

The funny thing is that C++ sort of has a flexible string class which uses system-dependent conversions, std::filesystem::path

It's super convenient for working with Windows file APIs, but everything else is still a pain.

Yep. There are still some annoyances like invalid UTF-16, so maybe there is more is needed than just the basic transformations — some precise handling or parameters specifying how to handle these edge cases could be necessary.

Handling UTF-8 versus WTF-8, and properly round-tripping these representations, is probably a bigger issue than graphemes and canonical forms for most users.

Also unfortunately the web browser. Javascript strings are UTF-16 (Well, UCS2).

This used to be the case when JS was created. However, I don't think that this is true anymore. It's definitely not under the hood. Nor, if I recall correctly, if you attempt to convert them from/to typed arrays. Maybe if you use `fromCharCode`?

JS strings are exposed as UTF-16 in the language — surrogate pairs have length 2 and you can index into half of them.

A JavaScript string is conceptually an array of 16-bit unsigned integers (https://tc39.es/ecma262/multipage/ecmascript-data-types-and-...). String indexing and substring operations treat strings this way, and they couldn't change the model easily without breaking code. If you index a string containing an emoji with a code point greater than U+FFFF, you get string containing an unpaired surrogate code unit, which makes the string UCS-2, not UTF-16. I don't know how it's implemented memory-wise in the various JavaScript engines and am not familiar with typed arrays.

> It's definitely not under the hood.

Really? I'd love to see some references to that.

I know there's some operations to convert a javascript string to UTF8, but everything which measures a string counts using 2-byte UCS2 items. For example, array-style indexing, string.length, str.charCodeAt(), split(), slice(), indexOf(), etc.

Also lots of DOM APIs use those weird string lengths too. For example, input element selectionStart / selectionEnd fields measure the selection range by counting surrogate pairs, not characters.

> Really? I'd love to see some references to that.

I remember that SpiderMonkey shifted to multiple encodings around 2019 and v8 had done that earlier.

Source: I was part of the SpiderMonkey team when the shift happened.

Reply on news.ycombinator.com