What every software developer must know about Unicode in 2023
tonsky.me
tonsky.me
The truth of the matter is that there are several different definitions of "character", depending on what you want to use it for. An extended grapheme cluster is largely defined on "this visually displays as a single unit", which isn't necessarily correct for things like "display size in a monospace font" or "thing that gets deleted when you hit backspace." Like so many other things in Unicode, the correct answer is use-case dependent.
(And for this reason, String iteration should be based on codepoints--it's the fundamental level on which Unicode works, and whatever algorithm you want to use to derive the correct answer for your purpose will be based on codepoint iteration. hsivonen's article (https://hsivonen.fi/string-length/), linked in this one, does try to explain why extended grapheme clusters is the wrong primitive to use in a language.)
I'm sorry, but I fail to see how "This visually displays as a single unit" could ever differ from "Display size in a monospace font" or "Thing that gets deleted when you hit backspace".
Some clusters are going to be multiple characters wide.
> thing that gets deleted when you hit backspace
Some clusters are meant to be composted of multiple keystrokes and a natural editing experience would allow users to delete the last stroke.
Look into how Korean works.
As for "display size in monospace font", emojis and CJK characters are usually two units wide, not one (although, to be honest, there's a fair amount of bugs in the Unicode properties that define this).
[1] https://android.googlesource.com/platform/frameworks/base/+/...
* Coding ligatures often display as a single glyph (maybe occupying a single-width character space, or maybe spread out over multiple spaces), but are composed of multiple glyphs. The ligature may "look" like a single character for purposes of selection and cursoring, but it can act like multiple characters when subject to backspacing.
* Similarly, I've seen keyboard interfaces for various languages (e.g., Hindi) where standard grapheme cluster rules bind together a group of code points, but the grapheme cluster was composed from multiple key presses (which typically add one code point each to the cluster). And in some such interfaces I've seen, the cluster can be decomposed by an equal number of backspace presses. I don't have a good sense of how much a monospaced Hindi font makes sense, but it's definitely a case where a "character" doesn't always act "character-like".
For example, when == is written, connect them to be a 2 column wide = instead.
Or when === is written, display a three column equals sign, but it's three bars instead of two.
Seems like the right answer for codepoints vs graphemes, unfortunately, is dependent on the context.
edit: emojis are filtered by HN
I guess it's more ambiguous for some languages that can have long ligatures though.
Where some layouts may require this method for some characters, another keyboard layout may have the same character on a dedicated key.
The program receives the combined character as one unit, and does not need to be aware of different keyboard layouts.
Which ones? At least the French and German ones don’t work like that: there is no composing, just separate keys for all the characters with diacritics that appear in the language.
It's worth mentioning of course that there are no-dead-keys variants of the keyboard layout but this has been pretty much the norm on Windows since the 1990s I think.
That being exactly the way “floating diacritics” in ISO 2022 (or properly one of its Latin encodings, T.51 = ISO 6937) work, amusingly. I wonder which came first. (Yes, I know that a<BS>` came first, the ASCII spec even says that this should give you an accented character IIRC. Or perhaps it was one of the other “don’t call it ASCII” specs—ISO 646? IA5?..)
A美C
Similarly, for composed characters like say the ligature “ff”, you may want to backspace as if it was two “f”s (which logically it is, and decomposes to in NFKD normalization).
Latin: Katakana
Full width: カタカナ
Half width: ｶﾀｶﾅ
Discussion on HN: https://news.ycombinator.com/item?id=31858311
𒐫
𒈙
꧅
[1] Relevant specifications: https://unicode.org/reports/tr29/#SpacingMark and https://unicode.org/reports/tr29/#GB9a
It reminds me a little of Open Street Map's inconsistent administrative hierarchies ("states", "countries", "counties", etc. being represented at different administrative "levels" in their hierarchy for each geographical area), and how that hinders consistency in styling- font size, zoom levels, etc. being generally applied by level.
My impression before was it would be considered a single "entity" (not the same as a roman-alphabetic word, but not a character either) containing 3 characters.
Like a way to query what should be treated like a single "symbol" when selecting text? Basically something that could help out users making simple text-editors. There are so many bad implementations out there that does it incorrectly so there must be some tools/libraries to help with this? Not only for actual applications but for people making games as well where you want users to enter names, chat or other text. Not all platforms make it easy (or possible) to embed a fully fledged text editing engine for those use-cases.
I can imagine that typing a multi-code-point character manually by hand would allow the user to undo their typing mistake by a single backspace press when they are actively typing it, but after that if you return to the symbol and press backspace that it would delete the whole symbol (grapheme cluster).
For example if you manually entered the code points for the various family combination emojis (mother, son, daughter) you could still correct it for a while - but after the fact the editor would only see it as a single symbol to be deleted with a single backspace press?
Or typing 'o' + '¨' to produce 'ö' but realizing you wanted to type 'ô', there just one backspace press would revert it to 'o' again and you could press '^' to get the 'ô'. (Not sure that is the way in which you would normally type those characters but it seems possible to do it with unicode that way).
As for selection and cursor movement, grapheme clusters would seem to be the correct choice. Same for Delete. An editor may also support an “exploded” view of separate characters (like WordPerfect Reveal Codes) where you manipulate individual characters.
On further reflection, probably the best starting point for string editing on backspace is to operate on codepoints, not grapheme clusters. For most written languages, the various elements that make up a character are likely to be separate codepoints. In Latin text, diacritics are generally precomposed (I mean, you can have a + diacritic as opposed to precomposed ä in theory, but the IME system is going to spit out ä anyways, even if dead keys are used). But if you have Indic characters or Hangul, the grapheme cluster algorithm is going to erroneously combine multiple characters into a single unit. The issue is that the biggest false positive for a codepoint-based algorithm is emoji, and if you're a monolingual speaker whose only exposure to complex written scripts is Unicode emoji, you're going to incorrectly generalize it for all written languages.
This is a good example because in German I would expect 'o' + '¨' + <delete> to leave no character at all while in French I would expect 'e' + '`' + <delete> to leave the e behind because in my mind it was a typo.
The rendering of brahmic- and arabic-derived scripts makes these choices even more interesting.
I'm not sure that ever really makes sense: it would be a misnomer if "backspace" didn't bring you "back" some amount of horizontal "space," I reckon. This logic holds up not only for cases like ö and emoji (where I'd hope the whole grapheme disappears), but also for scenarios like if one types <f><i> and an <fi> ligature appears, where I'd hope only the <i> disappears: that's fine because you are still going back some space.
If the key ever gets repurposed from "backspace" to "undo" then I would agree that it should step back to the previous state with as much granularity as possible.
Dieresis (Greek word we use in English) is a phenomenon where two vowels don’t merge together to make a new sound. Co-operate, coöperate, or cooperate. You could also shove a glottal stop in there but this is much less extreme. The diaresis marker isn’t used much in the USA. Another common example is naïve.
Umlaut (German word) is the opposite phenomenon: the first vowel sound is inflected by the following, so in German Apfel (apple, pronounced UPfell) becomes Aepfel (apples, EPPfell), instead commonly written Äpfel instead.
People usually refer to the marks by the term in their own language (umlaut, diaresis, caron, accent…). I say Umlaut in German (which doesn’t mark the phenomenon, you just have to know) and diaresis in English though I’m talking about the same symbol.
But diacritics are used in no systematic way in different languages anyway (this is true of the letters themselves too: letters like S, W, V and Z have different pronunciations in English and German, for example). Ppl just needed ways to say “the sound here is similar to the sound of this letter but not really the same” or “when you say this letter also do something else” (like, say, add stress for languages that have that).
English mostly dispenses with them, sometimes adding a letter (g vs gh) but mostly having a very casual relationship with the sound at all. You simply have to know that cooperate isn’t pronounced like chicken coop. I personally think that’s a good thing.
It appears there's a few keyboard layouts and language options:
French (Canada) - Canadian French
French (Canada) - Canadian Multilingual Standard
French (France)
etc
And then I think it intersects with what keyboard you actually have.
Some of these layouts maybe are designed to enable you to type in French characters on a non-French keyboard? Not sure.
Furthermore, you shouldn't assume that there is any relationship between how unicode constructs a combined character from codepoints with how that character is typed, even at the level of typing you're not typing unicode codepoints - they're just a technical standard representation of "text at rest", unicode codepoints do not define an input method. Depending on your language and device, a sequence of three or more keystrokes may be used to get a single codepoint, or a dedicated key on keyboard or a virtual button may spawn a combined character of multiple codepoints as a single unit; you definitely can't assume that the "last codepoint" corresponds to "last user action" even if you're writing a text editor - much of that can happen before your editor receives that input from e.g. OS keyboard layout code; your editor won't know whether I input that ö from a dedicated key, a 'chord' of 'o' key with a modifier, or a sequence of two keystrokes (and if so, whether 'o' was the first keystroke or the second, opposite of how the unicode codepoints are ordered).
Korean packs syllables into Han-script-like squares, but they are unmistakably composed of alphabetic letters, and are both typed and erased that way (the latter may depend on system configuration), yet the NFC form has only a single codepoint per syllable (a fortiori a single grapheme cluster). Hebrew vowel markings are (reasonably) considered to be part of the grapheme cluster including their carrier letter but are nevertheless erased and deleted separately. In both of those cases, pressing backspace will erase less than pressing shift-left, backspace; that is, cursor movement and backspace boundaries are different.
There are IIRC also scripts that will have a vowel both pronounced and encoded in the codepoint stream after the syllable-initial consonant but written before it; and ones where some parts of a syllable will enclose it. I don’t even want to think how cursor movement works there.
Overall, your suggestion will work for Latin, Cyrillic, Greek, and maybe other nonfancy scripts like Armenian, Ge’ez, or Georgian, but will absolutely crash and burn when used for others.
Backspace in general seems to be a very difficult problem because of subtly incompatible expectations depending on the context, as 'undo last input' when you're typing new text, and 'delete previous symbol' if you're editing existing text.
On the other hand, a ligature shouldn't be deleted entirely with just one backspace. It's two letters after all, just connected.
So how do we distinguish when codepoints are separate graphemes, and when they constitute a single grapheme? Based on if they they can still be recognized as separate within the glyph? If they combine horizontally vs vertically (along the text direction or orthogonal?) What about e.g. "¼" - are those 3 graphemes? What about "%" and "‰"? What about "&" ("et" ligature)? It seems you can't run away from being arbitrary…
"Length" is a meaningless query because of this, but you might want to default to whatever approximates width in a UI label, so that's grapheme clusters. Using codepoints mostly means you wish you were doing bytes.
Strange thing to say: Swift String count property is the count of extended grapheme clusters. The documentation is explicit:
> A string is a collection of extended grapheme clusters, which approximate human-readable characters. [emphasis in original]
The parallels of string length with the phrase "How long is a piece of string?"[0] make this apparent/amusing. I'm sure I'm not the first person to think that.
[0]: https://en.wiktionary.org/wiki/how_long_is_a_piece_of_string
1. To save storage space or avoid pathological input, they want to limit the "length" of text input fields. E.g., not allow a name to be 4 KB long
2. To fit something on screen
Developers mostly used to western languages can approximate both with "number of letters", but the correct answers are
For 1. Limit to bytes to avoid people building infinite zalgo characters, but be intelligent about it - don't just crop the byte array not taking into account graphemes.
For 2. This sucks, especially for the web, but the only really correct answer here is to render it and check.
Did I miss any other common cases?
"for loop counts" depends on what you're doing in the loop...
https://github.com/electron-userland/electron-builder/issues...
Just wanted to let you know in case it's a gotcha you might not be aware of that might help you out if you run into similar problems with some of your customer builds.
The bug you linked to is a bit confusing, it seems to be a bug in electron-builder (or node-gyp), not Conveyor?
Is there a canonical source for this part, by the way? Xi copied the logic from Android[1] (per the issue you linked downthread), which is reasonable given its heritage but seems suboptimal generally, and I vaguely remember that CLDR had something to say about this too, but I don’t know if there’s any sort of consensus on this problem that’s actually written down anywhere.
Why not offer both and be clear about it? Rather than just "length", why not call them code points? The Python docs for "len" which can be called on a unicode string say "Return the length (the number of items) of an object.". It doesn't look like a clear and easy to use API to me.
for c in "Hello":
pass
if word[0] == 'H':
pass
This would have been an extremely controversial suggestion when python3 came out to say the least.
Codepoints is a natural way of defining unicode strings in python, and it mostly works the way you expect once you give it a bit of thought. It is lower level than, say, grapheme clusters, but its more well defined and it provides the proper primitives for dealing with all use cases.
This leads you to the problem where you'll get different results iterating over
n a ï v e
vs
n a ̈ i v e
And I can't see how that's ever going to be a useful outcome.
If you normalize everything first, then you can sidestep this to some degree, but then in effect your normalization has turned codepoint iteration into grapheme iteration for most common Latin-script text characters.
> Why does the fi ligature even have its own code point? No idea.
On of the principles of Unicode is round trip compatibility. That is you should be able to read in a file encoded with some obsolete coding system and write it out again properly. Maybe frob it a bit with your unicode-based tools first. This is a good principle, though less useful today.
So the fi ligature was in a legacy encoding system and thus must be in Unicode. That's also why things like digits with a circle around them exist: they were in some old Japanese character set. Nowadays we might compose them with some zwj or even just leave them to some higher level formatting (my preference).
Edit ①: seems the answer is not with the default iOS keyboard, but possible to paste it and perhaps possible with a third party keyboard that I'm not keen on trying (unless I hear of a keyboard that's both genuinely useful / better than default, and that doesn't send keystrokes to the developer - though I can't remember if the latter is even a risk on iOS, better go search about that next..)
Today, it’s buggy $5 abandonware that only makes me sad when I am reminded of it.
EDIT: Here[1] is a blog post that claims it's still the best keyboard in 2023. I actually might give it another shot... Not holding my breath though.
*EDIT: Looks like another dedicated fan has actually taken it upon themself to revive the project, under the new name Keyboard71[2].
[0] https://apps.apple.com/us/app/nintype/id796959534
Really not looking to burn another $5, I’d greatly appreciate any thoughts/concerns at all.
We should be careful: someone on HN could write a proposal that they should be considered pre composed forms that should also have an un-composed sequence… so there could in future be not just 1 in a circle but 1 ZWJ circle, circle ZWJ 1 all considered the same…I can imagine some HN readers being pranksters like that.
This implies that they're obsolete, but they're not -- they're still in very common use today. You can type them in Japanese by typing まる (maru, circle) and the number, then pick it out of the IME menu. Some IMEs will bring them up if you just type the number and go to the menu, too. :)
① Draw some circles
② Draw the rest of the owl
Commentary: ① is simple but ② is masking many complex steps that are necessary to draw an owl.
Most of the pre-composed latin ligatures are generally from EBCDIC codepages. People in the ancient Mainframe era wanted nice typesetting too, but computer fonts with ligature support were a much later invention.
You can see fi and several others directly in EBCDIC code page 361:
https://en.wikibooks.org/wiki/Character_Encodings/Code_Table...
Actually in German (at least) ä, ö and ü really are actually ligatures for ae, oe, and ue -- the scribes started to write the E's on their sides above the base letters, and over time the superscript "E"s became dots or dashes. Often they are described the other way around: "you can type oe if you can't type ö." That's what my kid was told in school!
But Ö and ß aren't really part of the alphabet in German, while, say, in Swedish, ä and ö became actual letters of the alphabet. English got W that way too.
By contrast, my French dictionary has énorme nestled between enorgueillir and enquiérir. (Looking for an example does underscore some of the patterns in the language: page after page of ét~ with only a few et~ and one êt~ among them; pages of ex~ with no éx~ at all.)
Also, though it's hardly authoritative, my kids' school taught English through immersion from grade 1 too, and both German and English teachers said "same alphabet".
As bmicraft pointed out, even in that wikipedia chart those inflected letters are spaced apart from the others. Yes, they are letterforms, but not part of the "alphabet" -- they don't even have a sorting like the Swedish Ä or W do.
And you can switch in running text from using the marker for umlaut (dots or bar, not semantically dieresis) or a normal "e" without anyone blinking. There's no problem reading a Swiss book even though ß refuses to cross the border. Though I personally prefer to read Äpfel and Bär rather than Aepfel and Baer, really, they are the same.
On a somehow related side note, I read that "&", which is derived from a ligature of the Latin conjunction "Et" (meaning "and"), was named "ampersand" in English as a mondegreen for "and per se and" as it used to be placed at the end of the English alphabet recitation.
They still ꜩ on some German street signs. I can't find ch in Unicode though (could just be my old eyes).
I have my music teacher's German schoolbook from around 1915 and it lists them all as letters (the whole book is in Fraktur). I have various old books in Fraktur and once could read them. I imagine that if I sat down and tried to read one it would come back, but at the moment I have to thin a little to read just the titles!
Replace "digits with a circle around them" with "emojis" and that's also true.
eg https://www.npr.org/2023/05/02/1173418268/loneliness-connect...
Probably not the greatest during a hacker news hug of death, but if I read that article some other time and saw one of the moving pointers, I would think it was really cool.
[0] WS connection is on `wss://tonsky.me/pointers?id=XXXXXX&page=/blog/unicode/&platform=XXX`
I really enjoyed the tongue in cheek design. I think every modern browser either allows you to turn on reader mode (especially on mobile) or just turn off CSS. This particular article works excellently even in w3m.
:D
> I really enjoyed the tongue in cheek design. I think every modern browser either allows you to turn on reader mode (especially on mobile) or just turn off CSS. This particular article works excellently even in w3m.
Firefox Focus on mobile does not have Reader Mode.
I should also add that the night mode won't be as fun on mobile either, just checked and I don't think it works with pointer events for the effect
The decision to design a serious (read: not-tongue-in-cheek) topic with these "quirky" tricks sucks, JMHO.
I can not imagine manually approving JS for every site. And again doing the inverse and have noscript installed to deny one website a year does not seem to be worth it for me. In this case I can also just use a adblock rule to block that specific script or all .js files from the domain I guess. So I really no not need NoScript.
For those sites that do need JS, NoScript's whitelist feature makes it quick & easy to fix. The first time I visit a new site, if it is obviously broken, then I whitelist the main domain. If that doesn't fix it, then I whitelist a couple likely-looking domains (often sites import JS from similar domains, or from common library domains). That's enough to get probably 90+% of websites working, while still leaving most garbage JS disabled. The remaining ~10% of websites that need a dozen domains whitelisted are probably not worth visiting anyway, so I just move on at that point. Or NoScript even lets you temp-whitelist everything for a given tab and just put up with the misery to get whatever I need from that one site. Since the whitelist persists forever, and I don't visit hundreds of different websites every day, after some time it becomes pretty rare that I need to whitelist more than one or two things per day.
You maintain an adblock blacklist, I maintain a NoScript whitelist. Not so different :)
As others have pointed out, reader mode works as expected.
2. I don't think it's a very good joke to post long-form content on your blog with the expectation that it's basically unreadable without a reader mode.
> The fact that people seem to take it seriously says something about the contemporary state of webdesign :)
Mind expanding and what it says exactly about contemporary web design?
Whether I take it seriously or not doesn't change the fact that it's still damn hard to read anything.
The same as when political satire is indistinguishable from actual politics. It means that the real things has sort of become a joke itself.
not even a proper flashlight. it updates when the mouse moves, so you're SOL if you scroll on desktop.
To each their own.
> It made me close the browser tab.
If you can't handle refreshing or merely clicking it again, that's you having a problem, not the site having a problem.
No, it's the site's problem. The contrast between the blinding radioactive yellow background and the font is eye straining and doesn't meet the WCAG standards for accessible text. And the dark mode is unusable. The joke would be funnier if there was a real dark mode or if the light mode was readable.
> doesn't meet the WCAG standards for accessible text.
Oh, which part of the standards? When I punch #000000 on #FDDB29 into contrast checkers I get good results.
The single quote isn't as good but all the rest has those colors.
.pointer.l {
background-color: green;
}Ideally - the "English-World" locale is supposedly meant for us, cosmopolitans. It's included with Windows 10 and 11.
Practically, as "English-World" was not available in the past (and still wasn't available on platforms other than Windows the last time I checked), I have always been setting the locale to En-US even though I have never been to America. This leads to a number of annoyances though. E.g. LibreOffice always creates new documents for the Letter paper format and I have to switch it to A4 manually every time. It's even worse on Linux where locales appear to be less easy to customize than in Windows. Windows always offered a handy configuration dialog to granularly tweak your locale choosing what measures system you prefer, whether your weeks begin on sundays or mondays and even define your preferred date-time format templates fully manually.
A less-spoken about problem is Windows' system-wide setting for the default legacy codepage. I happen to use single-language legacy (non-Unicode) apps made by people from a number of very different countries. Some apps (e.g. I can remeber the Intel UHD Windows driver config app) even use this setting (ignoring the system locale and system UI language) to detect your language and render their whole UI in it.
> English (USA), English (UK)
This deserves a separate discussion. I doubt many English speakers (let alone those who don't live in a particular anglophone country) care to distinguish between English dialects. To us presence of a huge number of these (don't forget en-AU, en-TT, en-ZW etc - there are more!) in the options lists brings only annoyance, especially when one chooses some non-US one and this opens another can of worms.
By the way I wonder how do string capitalization and comparision functions manage to work on computers of people who use both English and Turkish actively (Turkish locale distinguishes between dotted and undotted İ).
There's the English (Denmark) locale for that on some platfoms.
https://unix.stackexchange.com/questions/62316/why-is-there-...
It's worthwhile purely for the sake of autocorrect/typo highlighting in text-editing software. I don't miss the days of spelling a word correctly in my version of English but still being stuck with the visual noise of red highlighting up and down the document because it doesn't conform to US English.
It's sometimes not a trivial distinction either, for example I've heard of cases where surprised British redditors have found themselves banned from American subreddits for being homophobic when they were actually talking innocently enough about cigarettes!
> This deserves a separate discussion. I doubt many English speakers (let alone those who don't live in a particular anglophone country) care to distinguish between English dialects.
While that is generally (though not always) true, I would assume it's really a stand in for the much more relevant zh locales.
It is also rather relevant to es locales (america spanish has diverged quite a bit from europe spanish hence the creation of es-419), definitely french (canadian french, to a lesser extend belgian and swiss), and german (because swiss german). And it might be relevant for ko if north korea ever stops being what it is.
I mainly type in English but occasionally other languages - I use a combination of Mac & Linux - macOS has an (off-by-default but enable-able) lang-changer icon in the tray that is handy enough, but still annoying to have to toggle. Linux is much worse.
Mac also has quite a nice long-press-to-select-special character that at least makes for accessible (if not efficient) typing in multiple languages while using an English locale. Mobile keyboards pioneered this (& Android's current one even does simultanous multi-lang autocomplete, though it severely hurts accuracy).
---
> I doubt many English speakers care to distinguish between English dialects.
I think you'll find the opposite to be true. US English spellings & conventions are quite a departure from other dialects, so typing fluidly & naturally in any non-US dialect is going to net you a world of autocorrect pain in en_US. To the extent it renders many potentially essential spelling & grammar checkers completely unusable.
While in my default (English) layout, it is easy enough to add in accents other characters using the compose key (right alt). So right-alt+'+a = á or right-alt+"+u = ü. I much prefer this over the long press as I can do it quickly and seamlessly without having to wait on feedback. Granted, it is not as discoverable, but once you are comfortable, it in my opinion is a better system.
Not for me!
Right-alt+'+a = â
Right-alt+'+u = û
I don't find it "unusable", though. I could change it back to en_US, but it has actually been interesting to see all of my American spellings flagged by autocorrect. Each time I write authorize instead of authorise it is an act of stubborn group affinity!
As far as the written, formal language is concerned, English really has only three dialects: US American, Canadian, and everywhere else. There are some other subtle differences (such as "robots" for traffic lights in South Africa, or "minerals" for fizzy drinks in Ireland¹), but that's pretty much it.
¹ Yes, this isn't just slang in Ireland: the formal, pre-recorded announcements on trains use it: "A trolley service will operate to your seat, serving tea, coffee, minerals and snacks." The corresponding Irish announcement renders it mianraí. Food service on trains stopped during covid and has not yet resumed, so I'm working from distant memory now.
This is true, but I don't see why the "formal" qualifier is needed here :) There are much more than 3 dialects of English, both written & spoken.
Especially there's a fair few extremely common notable differences in (casual, written) Irish English: the word "amn't" (among other less common contractions), the alternative present tense of the verb "to be" (i.e. "do be"), various regional plurals of "you", and - perhaps the most common - prepositional pronouns, etc. etc.
There's plenty of regional variants just within the US, but "en_us" covers the whole country.
I guess the simple answer to that is: how much interest is there in maintenance. I don't think there's any compelling reason not to create something: if there's insufficient interest in maintenance that's an imperfect but reasonable proxy for utility.
I'm not aware of any maintained en_US_Xyz languages but it might be pretty cool if someone started. There's precedence in a few other languages, like no_NO_NY, zh_Hans_HK, etc.
> I doubt many English speakers (let alone those who don't live in a particular anglophone country) care to distinguish between English dialects. To us presence of a huge number of these (don't forget en-AU, en-TT, en-ZW etc - there are more!) in the options lists brings only annoyance, especially when one chooses some non-US one and this opens another can of worms.
Well, you just explained what this plethora of options is about. It's not just about how you spell flavor/flavour. It's a lot of different defaults for how you expect your OS to present information to you. Default paper size, but also how to write date and time, does the week start on Monday, Sunday, or something else, etc.
I think you'd be surprised how many english (UK) people will get pissed off when their spell-checker starts removing the "u" from colour or flavour, or how many English (US) people get pissed off when the spellchecker starts suggesting random "u"s to words.
additionally to that, locale isn't just about language. English (US) and English (UK) decides whether your dates get formatted DD-MM-YY or MM-DD-YY, whether your numbers have the thousands broken by commas or spaces, and a host of other localization considerations with a lot more significance than just the dialect of english.
Perhaps you're an old-timer? I worked in the city in the early 80s; lunch in the pub was routine, and sometimes required. By the end of the 80s, that was at best frowned on. Over the last 20 years, having alcohol on your breath after lunch would have been a disciplinary issue, unless you were entertaining a client, at least in the places I worked.
ı I
I symphatize with people who get this wrong. (I just saw some YouTube video have a title TÜRKIYE in a segment)
Even google keyboard can't seem to distinguish between I and İ. When I type "It", it suggests "İt's" which is quite pathetic.
I can understand that the conflation of spelling, word choices, time and date formatting, default paper sizes, measurement units, etc, etc, is convenient, and works a lot of the time, but it really doesn't work for me at all.
That said, I appreciate that I occupy a very small niche.
Most people in the UK care - a population nearly twice that of California, and larger than the native speakers of any non-top-20 language. If you care enough to support e.g. Italian you should support en_UK.
I definitely do. The biggest difference, as everyone else, has pointed out is the US vs UK spellings.
Realistically, though, beyond that country is a poor indicator for everything else. I want to use DD/MM/YYYY date format in English, but DD.MM.YYYY date format in German. I want to use $1,000 in English, but 1.000 $ in German. This isn't dependent on the country I live in, this is dependant on a combination of a country and language - that could be the country I'm living in, or the country I grew up in (mostly US date format vs not), and it's either the language I'm actively typing in, or the language of the document I'm reading, or the language I'm thinking in (but a computer can't exactly handle that).
Trying to guess the correct combination is tricky, especially if a document is in two languages (e.g., a quotation), and users are lazy and won't switch their IME unless they have to.
What this means in slightly more practical terms is that setting a single "locale" for my device doesn't make sense, but rather I should be able to choose a locale per language (or possibly spelling preferences by language and formatting options by language as separate choices). I'd then pick a language to use the device in, and it would use that languages locale, and tell apps that this language is the preferred language. If an app doesn't provide my preferred language, pull the preferred locale from settings for a language it does support, otherwise use the default set by developers. For some apps, it's a bit more complex, particularly if I'm creating content. GMail or Office would be two good examples, where the UI language might be in English, but the emails or documents are in German, or a combination of German and English.
Even then, I'm sure there are people who need something even more flexible than that.
At the moment, if I set my language to English but my Country to Germany on my iPhone, for example, things occasionally get confused. My UK banking app, for example, pulled the decimal separator from my locale settings for a while and then refused to work because "£9,79" (or whatever it was) isn't a valid amount of money, and I couldn't see a way to fix that without switching my Country in the phone settings. I imagine they fixed it by ignoring my configured locale and always using en-GB, thus defeating the whole point of a locale in the first place.
So yeah, these days it's fairly common to not have a single "locale" that you work in - it's quite possible to want to use two or more but nothing is really set up to handle that well.
This leads to absolutely horrendous rendering of Chinese filenames in Windows if the system locale isn’t Chinese. The characters seem to be rendered in some variant of MS Gothic and it’s very obviously a mix of Chinese and Japanese glyphs (of somewhat different sizes and/or stroke widths IIRC). I think the Chinese locale avoids the issue by using Microsoft YaHei UI.
Why would a frontend developer demonstrate their ability to do frontend programming on their personal, not altogether super-serious blog? I meant that rhetorically but it's a flex. I agree, not the best design in the world if you're catering for particular needs, but simple and fun enough. You should check out dark mode.
In that vein, I think it's okay if we let people have fun. That might not work for everyone, but why should we let perfect be the worst enemy of fun?
because it shows that they don't understand important design aspects
while it doesn't really show off their technical skills because it could be some plugin or copy pasted code, only someone who looks at the code would know better. But if someone care enough about you to look at your code you don't need to show of that skill on you normal web-site and can have some separate tech demo.
> okay if we let people have fun
yes people having fun is always fine especially if you don't care if anyone ever reads your blog or looks at it for whatever reason (e.g. hiring)
but the moment you want people to look at it for whatever reason then there is tension
i.e. people don't get hired to have fun
and if you want others to read you blog you probably shouldn't assault them with constant distractions
Living by that motto is hugely self-destructive.
Creative expression allows us to push ourselves, both in what we think we can do, and often the technical aspects about how we do it too. Even if the idea doesn't stick, you've tried something new.
In a world of Tailwinds and Bootstraps and the same five templates copied again and again and again, let's celebrate the people willing to push things and learn from their inevitable but ultimately valuable mistakes. And let's have some fun along the way.
It's revenge against anyone with certain kinds of visual impairments and/or concentration issues because the ex-spouse of the author which turned out to be a terrible person had such.
(sarcasm try 2)
It's revenge against anyone using JS on the net with the author trying to subtle hint that JS is bad.
(realistic)
It's probably on of:
- the website is a static view of some collaborative tool which has that functionality build in by default
- some form of well intended but not well working functionality add to the site as it was some form of school/study project, in that case I'm worried about the author suffering unexpected very much higher cost due to it ending up on HN ...
1. Fun. Modern internet is boring, most blog posts are just black text on white background. Hard to remember where you read what. And you can’t really have fun without breaking some expectations.
2. Sense of community. Internet is a lonely place, and I don’t necessarily like that. I like the feeling of “right now, someone else reading the same thing as I do”. It’s human presence transferred over the network.
I understand not everybody might like it. Some people just like when things are “normal” and everything is the same. Others might not like feeling of human presence. For those, I’m not hiding my content, reader mode is one click away, I make sure it works very well.
As for “unexpectedly ended up on HN”, it’s not at all unexpected. Practically every one of my general topic articles ends up here. It’s so predictable I rely on HN to be my comment section.
Then the second thought was: I should again start to block js by default as much as I can.
https://hn.algolia.com/?dateRange=all&page=0&prefix=false&qu...
The cursors will only be a problem during front page HN traffic. And the opt-out for people who care is reader mode / disable js / static mirror. Not sure if there's any better way to appease the fun-havers and the plain content preferrers at the same time. Maybe a "hide cursors" button on screen? I, for one, had a delightful moment poking other cursors.
Meanwhile, ironic irony is ironic: "Hey, idiots! Learn to use Unicode already! Usability and stuff! Oh, btw, here is some extremely annoying Javascript pollution on your screen because we are all still children, right? Har har! Pranks are so kewl!!!1!"
I disagree.
What is the "right" things is use-case dependent.
For UI it's glyph bases, kinda, more precise some good enough abstraction over render width. For which glyphs are not always good enough but also the best you can get without adding a ton of complexity.
But for pretty much every other use-case you want storage byte size.
I mean in the UI you care about the length of a string because there is limited width to render a strings.
But everywhere else you care about it because of (memory) resource limitations and costs in various ways. Weather that is for bandwidth cost, storage cost, number of network packages, efficient index-ability, etc. etc. In rare cases being able to type it, but then it's often us-ascii only, too.
defaults matter
as in they should things you can just use by-default without thinking about it
as swift is deeply rooted in UI design having a default of glyphs make sense
and as rust is deeply rooted in unix server and system programming utf-8 bytes make a lot of sense
through the moment your language becomes more general purpose you could argue having a default in any way is wrong and it should have multiple more explicit methods.
That time has passed. If you want to know the length of a string, you really should indicate what length type you mean.
It has distinct .chars .codes and .bytes that you can specify depending on the use case. And if you try to use .length is complains asking you to use one of the other options to clarify your intent.
my \emoji = "\c[FACE PALM]\c[EMOJI MODIFIER FITZPATRICK TYPE-3]\c[ZERO WIDTH JOINER]\c[MALE SIGN]\c[VARIATION SELECTOR-16]";
say emoji; #Will print the character
say emoji.chars; # 1 because on character
say emoji.codes; # 5 because five code points
say emoji.encode('UTF8').bytes; # 17 because encoded utf8
say emoji.encode('UTF16').bytes; # 14 because encoded utf16I was working on an app in Swift that needed full emoji support once. Team ended up writing our own string lib that stores things as an array of single-character Swift strings.
This was true while Swift was developing but it's been stable now for several years. At some point that complaint is no longer valid.
Rust was given as one of the examples and Rust's .len() behaviour is chosen based on three very reasonable concerns:
1. They want the String type to be available to embedded use-cases, where it's not reasonable to require the embedding of the quite large unicode tables needed to identify grapheme boundaries. (String is defined in the `alloc` module, which you can use in addition to `core` if your target has a heap allocator. It's just re-exported via `std`.)
2. They have a policy of not baking stuff that is defined by politics/fiat (eg. unicode codepoint assignments) into stuff that requires a compiler update to change. (Which is also why the standard library has no timezone handling.)
3. People need a convenient way to know how much memory/disk space to allocate to store a string verbatim. (Rust's `String` is just a newtype wrapper around `Vec<u8>` with restricted construction and added helper functions.)
That's why .len() counts bytes in Rust.
Just like with timezone definitions, Rust has a de facto standard place to find a grapheme-wise iterator... the unicode-segmentation crate.
It's a poor and misleading example for it is definitely not how 'é' is encoded in 99.999% of all the text written in, say, french out there (french is the language where 'é' is the most common).
'é' is U+00F9, one codepoint, definitely not two.
Now you could say: but it is also the two codepoints one. But that's precisely what makes Unicode the complete, total and utter clusterfuck that it is.
And hence even an article explaining what every programmer should know about Unicode cannot even get the most basic example right. Which is honestly quite ironic.
However, the author could have added a small note, e.g. "(Unicode normalization will be convered in a later section.)", to prevent knowledgable readers from rage quitting :)
Yikes, does it really deserve that much derision? They’re trying to standardize all written human language here. I think they’ve done a fantastic job. Pre-Unicode you had to worry about what code page a document had, and computers from different countries couldn’t interoperate. The work the consortium does is hugely important, and every decision has extremely complex tradeoffs. Composed characters makes a lot of sense, and there’s a lot of case to be made that it was the right call. The attitude of “this one thing I don’t like makes the whole thing a complete clusterfuck” is something I wish fewer engineers would have.
To my simple mind it had one job: allocate every grapheme a number (code point). Had it done that, the 1/2 of the article warning you about the difficulty of iterating and modifying code points would have disappeared.
But I guess it had a 2nd job: create a way of representing those numbers. The obvious way, u32, was difficult for ASCII users swallow as it quadrupled the space used for a string. The solution we settled on, UTF-8 didn't come form Unicode (or ISO). It came from Ken Thompson (the Ken Thompson, who created B, the predecessor of C), when he tired to make something workable for C.
Unicode was the entity that ballsed up both of those tasks. It was a fork of ISO 10646. It's main contribution over 10646 was UCS-2 - ie 16 bits per character. That decision was so bad it had to be abandoned. Later they introduced the grapheme clusters rather than allocating a separate code point for each variant. I have no idea why, as it makes the programmers task far harder. Maybe they ran out of code points. How could they possibly run out of code points, given U32 has 4 billion of them and UTF-8 could potentially have more? Because they had to kludge their way around the USC-2 mistake to create UTF-16, and it's limited 1 million.
Which leads us to the one thing in the article I disagree with:
> The only downside of UTF-16 is that everything else is UTF-8, so it requires conversion every time a string is read from the network or from disk.
No, that's not the only downside. There is one more: USC-2 / UTF-16 has the endianness problem. A 16 bit value needs two bytes to represent it, and you can write two bytes to storage in two ways - little endian or big endian. They didn't specify, so the same string can have two different representations on disk. They added the infamous BOM markers to distinguish between them.
I could go on, but colour me singularly unimpressed with this mob.
It never was. The earlier draft of ISO/IEC 10646 bears absolutely no resemblance with the current 10646 and Unicode (for example, the first character in ISO/IEC DIS 10646:1990 was 0x20202020, which I believe is mapped to a space U+0020). Unicode had a much better design compared to 10646 so the final 10646 was retrofitted to Unicode instead.
> It's main contribution over 10646 was UCS-2 - ie 16 bits per character. That decision was so bad it had to be abandoned.
UCS-2 was already in 10646 in the draft stage. It had an even worse mechanism than surrogate pairs: escape sequences from ISO/IEC 2022 to switch groups and planes (upper 16 bits of code point). The standardized UCS-2 doesn't have them because of the merger of then-16-bit Unicode.
> Later they introduced the grapheme clusters rather than allocating a separate code point for each variant.
That sounds like that Unicode initially allocated separate code points for each variant. They didn't, or rather couldn't. An easy example is a Latin character with combining marks. There are a lot of combining marks in existence, some even defined before Unicode (yes, it's not the Unicode invention!), and sometimes a single character can have multiple marks. So Unicode only gave separate code points for compatibility, and otherwise resorted to the normalization mechanism that understands how to handle such cases.
The concept of grapheme cluster naturally arises from the existence of normalization. Not in the strictest sense, but it can be thought as a closed set over normalization and concatenation, so that it roughly matches with user-perceived characters. So grapheme clusters were already there, only the precise algorithm was specified later.
> To my simple mind it had one job: allocate every grapheme a number (code point).
You can easily have more than 10M code points in this way. The current set of Hangul syllables, precomposed or not, is 125 * 95 * 138 = 1,638,750 characters. Latin characters with at most 3 combining marks (known to exist in the wild) would be probably in the same order of magnitude. Maybe now you can try, thanks to the computing power and all the information, but in 1990? Fat chance.
> Maybe they ran out of code points. How could they possibly run out of code points, given U32 has 4 billion of them and UTF-8 could potentially have more?
For last 20 years the rate was about 2,700 new code points per year. It would take more than 200 years to fill all other unassigned planes at this rate. And most "new" code points (in quantity) are for rare or ancient Han characters, which are technically unbounded but strongly bounded by existing ancient works and scholarly works to uncover them. I doubt there remain more than 100,000 potentially encodable Han characters.
> You can easily have more than 10M code points in this way. The current set of Hangul syllables, precomposed or not, is 125 * 95 * 138 = 1,638,750 characters. Latin characters with at most 3 combining marks (known to exist in the wild) would be probably in the same order of magnitude. Maybe now you can try, thanks to the computing power and all the information, but in 1990? Fat chance.
It can be made to work both ways. The current situation pushes the handling of compose points onto the application programmer. Every time he wants to index into a array of characters, maybe to handle backspace of the user pressing arrow keys, he's forced to handle composition. But your average programmer tasked with writing gathering some information from the web or creating an accounting package doesn't care about this stuff, so he's going to stuff it up 10 times out of 10. That why the sorts of problems illustrated original article are legion today.
The alternative is 10M code points as you say. But it doesn't have to be 10M real code points. Someone down the software stack, a piece of software could say "oh, this code point represents a composed grapheme, I'll break it down into it's parts". In fact "break it down into it's parts" might mean turn it into exact representation we have now.
The difference between the two alternatives is who has to do the work. With the composition approach, the font rendering library has it slightly easier but the application write has to do more work. In the 10M code points approach, the application writers job has been made easier, at the expense of the font rendering library coder job has become harder.
It seems pretty obvious to me which of those two approaches wins. There are literally orders of magnitude more end user applications than there are font rendering libraries out there, and what's more the font rendering library programmers are far more likely to very concerned about doing grapheme clustering right. So if you took the 10M code point approach, you would have saved the planet a lot of code, and got a better result to boot.
As for the rest - your correction that 10646 was the source of many of them problems is appreciated. But that doesn't alter the fact that from a programmers perspective the spec is far harder to implement at the business end than it should be. The problems started with USC-2 and it compounded form there. And as a consequence, we have a large number of font rendering bugs we could have escaped had the spec been done differently.
If my understanding is correct, your thesis is that Unicode should be hidden from application programmers as much as possible, much like the fact that GC hides memory management so to say. Not to say Unicode is bad or even shouldn't exist at all, but something like that it has to be abstracted away. This is a much more reasonable than most (quote-unquote) Unicode criticisms indeed. I'm not sure whether this is possible in the near future however, for reasons I'll work out here.
----
From perspectives of API consumers, most if not all programming languages have a suboptimal design for the human text. In fact the type name "string" itself is inappropriate, its name comes from the assumption that a human text is a string of symbols, which is not incorrect but not helpful either. A proper "human text" type (or a collection of them) should ideally be able to do the following:
- An abillity to contain additional linguistic informations like locales, grammatical genders or numbers if possible. Some can be guessed, some can be retrieved from external contexts (e.g. HTTP `Accept-Language`), some have to be retrived with a consent. Any text operation should retain them if the corresponding text is also retained.
- A language-aware formatter. For example `"Total: ${n} files"` should automatically change "files" to "file" when `n` is 1. Moreover, `"Total: ${n} ${objectName}"` should do the same if `objectName` is an English text "files". (This is why every text should retain linguistic informations!) Of course the format text should be translatable (say, to "파일 총 ${n}개" in Korean) and that shouldn't change the original code.
- Proper textual isolation. If my text is composed of multiple scripts or languages, they should not affect each other in any way, and should be displayed in the best way possible. For example a missing font should not give broken boxes; either the font should be downloaded on demand, or a note about missing font should be shown instead. Inserting RTL texts into LTR texts should not flip either of texts (unless it is required by the surrounding languages). Basically, no surprises even if you don't know about them.
- Situation-aware alternatives. Even after the formatting, a long text that doesn't fit into the UI should be shortened in the way that as many information is preserved as possible. For example the text "Nice to meet you, ImagineAVeryLongUserNameHere!" will be cut into "Nice to meet you, ImagineAVery..." today, but one should be able to turn this into "Hello, ImagineAVeryLongUserNam..." from the formatting layer.
None of these operations actually concern Unicode, but they are incredibly hard---if not impossible---to build. Most of them are at best fuzzily defined or often undefinable. So we are left with a number of localization and internationalization libraries which are ignored by most developers to say the least. A mere "string" type is a norm, a program has no idea about the text and proceeds with faulty assumptions, and users are so accustomed to bad text handling that they even don't expect much. If enough users complain there is a chance of improvements, but even that is done by a case-by-case basis.
---
Unicode sits at the level much lower than what I've imagined before. It is not even a component to build the human text. It is a component to build a string, that can be somehow used to build the human text if one is very careful. Most human text operations can't be done with Unicode alone.
For example, people argue that the number of "characters" in a single Emoji sequence (say, one mentioned in https://hsivonen.fi/string-length/) should be 1 and others don't make sense. This is meaningless because it will appear as a number of broken boxes if emoji fonts are not installed anyway (but not five, because it contains two default-ignorable code points). It matters what the number of "characters" is used for, and that's a whole point of the linked article. And the definition of user-perceived characters does vary over locales, so you can't count them without a linguistic information anyway.
You may still argue that Unicode algorithms are designed for the human text encoded in strings. That's a very nuanced argument, because one can also argue that they are the best effort approximation of human text operations for strings, in which case they are not the human text operations themselves. For example many languages have a case conversion operation over strings, with a varying degree of Unicode conformance (ASCII-only, simple fold, full fold, locale-dependent fold, title case, ...). But the case conversion itself is not the human text operation! Even assuming bicameral scripts, some texts are never capitalized (e.g. "McDonald" frequently capitalizes to "McDONALD", not "MCDONALD"). The human text operation, here full capitalization, needs much more than the Unicode case conversion algorithm.
Given this, it is a misguided effort to make strings more aligned with the human text, because it is not possible at all. As people frequently mistake strings as human texts however, the second best thing is to get rid of any string operation. Swift almost did this but retained a default grapheme view---I think it is actually worse given the instability of (extended) grapheme clusters over time, but also understand why they had to do that. [1] The third best thing is probably to stress that a string is not a human text, in the same way that a floating point number is not a real number. And the original post, in spite of some errors, did a good enough job in this regard.
[1] There is also a precedent of Raku's NFG (which dynamically allocates a negative code point for new grapheme clusters seen), but this is more or less an optimization of the graphemes view. The current Unicode has an infinite number of distinct grapheme clusters by design.
> You may still argue that Unicode algorithms are designed for the human text encoded in strings. That's a very nuanced argument, because one can also argue that they are the best effort approximation of human text operations for strings, in which case they are not the human text operations themselves. For example many languages have a case conversion operation over strings,
That is ... very nuanced indeed. Case folding does look to be difficult issue in it's own right, but it's not something I do a lot and besides even 20 years ago it was deferred to a library function (str.lower() or whatever the language provides).
The issue the original article correctly says every app trips over is neither nuanced, nor uncommon. It's boring stuff like handling backspace or left arrow cursor movement - stuff programmer have to do all the time, and that is difficult to put in a library. As a consequence they get it wrong over and over again. Unicode representing a grapheme with a single code point would fix that.
As you say I'm not sure there is a simple change you could make to Unicode that renders case folding or the other problems you describe easy. To me that's a strong hint it's not the right place to address those problems.
Maybe how it's input by the keyboard (I haven't checked) but not how it's output on the web or other documents.
Plenty of text goes through Unicode normalization which may convert it to two codepoints.
I'm inclined to agree - they took a big problem and made it into a big & complex problem.
I propose a fix - immediately depreciate all multi-codepoint graphemes and provide single codepoint alternatives. That we should need to normalise something a basic as our text leaves soooo much room for problems. And the idea that some graphemes encode magically colour (emojis) ... :rolling_eyes_emoji:
IBM and Apple in the early 1990's combined in Taligent to try to beat MS NT, but failed. But a lot of internationalization came out of that and was made open, at the perfect time for Java to adopt it.
Interestingly it wasn't just CJK but Thai language variants that drove much of the flexibility in early unicode, largely because some early developers took a fancy to it.
When you look at the actual variety in written languages, Unicode grapheme/code-point/byte seems rather elegant.
We're in the early days of term vectors, small floats, and differentiable numerics (not to mention big integers). Are lessons from the history of unicode relevant?
Raku's Str type has a `.chars` method that counts graphemes. It has a separate `.codes` method to count codepoints. It also can do O(1) string indexing at the grapheme level.
That Zalgo "word" example is counted as 4 chars, and the different comparisons of "Å" are all True in Raku.
You can argue about the merits of it's approach (indeed several commenters here disagree that graphemes are the "one true way" to count characters), but it feels lacking to not at least _mention_ Raku when talking about how different programming languages handle Unicode.
It's best to avoid making overly-general claims like this. There are plenty of situations that warrant operating on code points, and it's likely that software trying and failing to make sense of grapheme clusters will result it in a worse screwup. Codepoints are probably the best default. For example, it probably makes the most sense for programming languages to define strings as arrays of code points, and not characters or 16-bit chunks or an encoding, or whatever.
Sometimes editing wants to go inside clusters but that's not code-point based either.
I'd say that in a big majority of situations, code that is indexing an array with code points is either treating the indexes as opaque pointers or is doing something wrong.
Absolutely correct. All algorithms defined by the Unicode Standard and its technical reports operate on the code point. All 90+ character properties defined by the standard are queried for with the code point. The article omits this information and ironically links to the grapheme cluster break rules which operate on code points.
Very rarely will you be implementing those algorithms. And if you're looking at character properties, the article says you should be looking at multiple together, which is correct.
I don't see where the article mentions Unicode character properties [1]. These properties are assigned to individual characters, not groups of characters or grapheme clusters.
> Very rarely will you be implementing those algorithms.
True, but character properties are frequently used, i.e. every time you parse text and call a character classification function like "isDigit" or "isControl" provided by your standard library you are in fact querying a Unicode character property.
But you need to deal with the whole cluster. You can't just look at the properties on a single combining character and know what to do with it.
If the article's saying to iterate one cluster at a time, then if you're doing properties a direct consequence is that you should be looking at the properties of specific code points per cluster or all of them.
As others have pointed out in this thread, the article is full of the authors own personal opinions. The author suggests iterating text as grapheme clusters, but fails to consider that this breaks tokenizers, e.g. a tokenizer for a comma-separated list [1] won't see the comma as "just a comma" if the value after it begins with a combining character.
Right. Which means not just iterating by code point.
> The author suggests iterating text as grapheme clusters, but fails to consider that this breaks tokenizers, e.g. a tokenizer for a comma-separated list [1] won't see the comma as "just a comma" if the value after it begins with a combining character.
I don't think they're talking about tokenizers. It's a general purpose rule.
Also I would argue that a CSV file with non-attached combining characters doesn't qualify as "text".
If anything, your example is an illustration why it's dangerous to iterate over codepoints and not graphemes. Unless you're explicitly tranforming encodings to/from unicode, anything that processes text (not the encoding, but actual text content, like tokenizers do) - should work with graphemes as the basic atomic indivisible unit.
This is doubly wrong.
First, it conflates languages and writing systems. Malay and English use the same writing system but are different languages. American Sign Language is a language, but it has no standard or widely-adopted writing system. Hakka is a language, but Hakka speakers normally write in Modern Standard Mandarin, a different language.
Second, it's not that case that Unicode aims to encode all writing systems. For example, there are many hobbyist neographies (constructed writing systems) which will not be included in Unicode.
Doesn't the "private use space" technically satisfy this?
The problems with Unicode are mostly to do with internal inconsistencies and churn, problems that usually only affect programmers.
1. Different ways to encode the same visually indistinguishable set of characters as code points leading to normal forms, text that compares unequal even when it appears to be identical, the disastrous "grapheme clusters" concept and so on.
2. Many different ways to encode the same sequence of code points as bytes. Not only UTF-32/16/8 but also curiousities like "modified UTF-8".
3. Emoji. A fractal of disasters:
3.a. Updates frequently. Neither Unicode nor software in general was built on the assumption that something as basic as the alphabet changes every year. If you send someone an emoji, can their device draw it? Who knows! In practice this means messaging apps can't rely on the OS system fonts or text handling libraries anymore which is a drastic regression in basic functionality.
3.b. (Ab)uses composition so much it's practically a small programming language, e.g. flags are composed of the two letter country code spelled using special characters. People are represented as as generic person plus skin color patch, families are represented using composed individual people etc.
3.c. Meaning of a character is theoretically specified but can subtly depend on the font used, e.g. people use a fruit emoji in visual puns because of how it looks specifically on Apple devices, so a "sentence" can make no sense if it's rendered with a different font.
3.d. Unbounded in scope. There's no reason the Unicode committee won't just keep adding new pictograms forever.
3.e. Encoded beyond the BMP which in theory every correct program should handle but in practice some don't because nobody except a few academics used characters beyond it much until emoji came along.
3.f. Disagreement over single vs double width chars, can only know this via hard-coded tables, matters for terminals and code editors.
Some of these can potentially be cleaned up outside of the Unicode consortium in backwards compatible ways. You could have a programming language that automatically normalized strings to fully composed form when deserializing from bytes, and then automatically folded semantically identical code points together (this would be a small efficiency win for some languages too). You could campaign to build a consensus around a specific normal form, like how UTF-8 gained consensus as a transfer encoding. You could also define a fork of Unicode (using private use areas?) that allocates a single code point to the characters that are unnecessarily using composition today but don't yet have one and then just subset out the concept of composition entirely.
Emoji are a big problem. It's tempting to say that these should not be encoded as characters at all. Instead there could be a set of code points that define bounds that contain a tiny binary subset of SVG, enough to recreate the Apple pixel art somewhat closely. Emoji would always be transmitted as inlined vector art. Text rendering libraries would call out to a little renderer for each encoded glyph, using a fast fingerprinting algorithm to deduplicate the bytes to an internal notion of a character. To avoid wire bloat, text can simply be compressed with a pre-agreed zstd or Brotli dictionary that contains whatever images happen to be popular in the wild. At a stroke this would avoid backwards compat problems with new emoji, enabling programs working with text to be upgraded once and then never again, eliminate all the ridiculous political committee bike-shedding over what gets added, let apps go back to using system text support and get rid of the bajillion edge cases that emoji have spewed all over the infrastructure.
I've written unicode-aware software for over a decade, doing a wide variety of programs, and I've never had to bother with all that mess.
If I'm parsing strings I'm looking for stuff in the 7-bit ASCII range which maps neatly onto the Unicode representations, and so I just need to take care to preserve the rest.
The only trouble I've had is that a lot of programmers haven't learned, or don't get, that text encoding is a thing and that it needs to be handled.
So they'll hand me an XML they claim is UTF-8 encoded, except that XML header was just copypasta and the actual XML document is encoded in some other system encoding like Windows-1252. Or worse, a mix of both.
For site local, fec0::3. Yeah site-local is discouraged but you can still do it. Or you can slightly misuse fd00::3.
You only get those latter 16 hex characters if you explicitly don't want to choose addresses.
"Exponential explosion" is really putting it too strong; it's perfectly possible to just add ǿ and á̤ and a bunch of other things. The combinations aren't infinite here.
The problem with e.g. Latin script isn't necessarily that combining characters exist, but that there's two ways to represent many things. That really is just a "mess": use either one system or the other, but not both. Hangul has similar problems.
Devanagari doesn't have any pre-compose characters AFAIK, so that's fine.
That's really the "mess": it's a hodgepodge of different systems, and you can't even know which system to use a lot of the time because it's not organised ("look it up in a large database"), and even taking in to account historical legacy I don't think it really needed to be like this (or is even an unfixable problem today, strictly speaking).
At least they deprecated ligatures like st and fl, although recently I did see ij being used in the wild.
They certainly are. Languages are a creative space driven by the human imagination. Give people enough time and they'll build new combinations for fun or for profit or for research or for trying to capture a spoken word/tone poem in just the right sort of exciting way. You may frown on "Zalgo text" [1] (and it is terrible for accessibility), but it speaks to a creative mood or three.
The growing combinatorial explosion in Unicode's emoji space isn't an accident or something unique to emoji, but a characteristic that emoji are just as much a creative language as everything else Unicode encodes. The biggest difference is that it is a living language with a lot of visible creative work happening in contemporary writing as opposed to a language some monks centuries ago decided was "good enough" and school teachers long ago locked some of the creative tools in the figurative closets to keep their curriculum simpler and their days with fewer headaches.
We've got 150K assigned codepoints assigned, leaving us with 950K unassigned codepoints. There's truly massive amounts of headroom.
To be honest I think this argument is rather too abstract to be of any real use: if it's a theoretical problem that will never occur in reality then all I can say is: <shrug-emoji>.
But like I said: I'm not "against" combining marks, purely in principle it's probably better, I'm mostly against two systems co-existing. In reality it's too late to change the world to decomposed (for Latin, Cyrillic, some others) because most text already is pre-composed, so we should go full-in on pre-composed for those. With our 950k unassigned codepoints we've got space for literally thousands of years to come.
Also this is a problem that's inherent in computers: on paper you can write anything, but computers necessarily restrict that creativity. If I want to propose something like a "%" mark on top of the "e" to indicate, I don't know, something, then I can't do that regardless of whether combining characters are used, never mind entirely new characters or marks. Unicode won't add it until it sees usage, so this gives us a bit of a catch-22 with the only option being mucking about with special fonts that use private-use (hoping it won't conflict with something else).
Unicode can't get rid of the many precombined characters for a huge number of backward compatibility reasons (including compatibility with ancient Mainframe encodings such as EBCDIC which existed before computer fonts had ligature support), but they've certainly done what they can to suggest the "normal" forms in this decade should "prefer" the decomposed combinations.
> If I want to propose something like a "%" mark on top of the "e" to indicate, I don't know, something, then I can't do that regardless of whether combining characters are used
This is where emoji as a living language actually shines a living example: It's certainly possible to encode your mark today as a ZWJ sequence, say «e ZWJ %», though you might want to consider for further disambiguation/intent-marking adding a non-emoji variation selector such as Variation Selector 1 (U+FE00) to mark it as "Basic Latin"-like or "Mathematical Symbol"-like. You can probably get away with prototyping that in a font stack of your choosing using simple ligature tools (no need for private-use encodings). A ZWJ sequence like that in theory doesn't even "need" to ever be standardized in Unicode if you are okay with the visual fallback to something like "e%" in fonts following Unicode standard fallback (and maybe a lot of applications confused by the non-recommended grapheme cluster). That said, because of emoji the process for filing new proposals for "Recommended ZWJ Sequences" is among the simplest Unicode proposals you can make. It's not entirely as Catch-22 on "needs to have seen enough usage in written documents" as some of the other encoding proposals.
Of course, all of that is theory and practice is always weirder and harder than theory. Unicode encoding truly living languages like emoji is a blessing and it does enable language "creativity" that was missing for a couple of decades in Unicode processes and thinking.
Yes, and that only makes things worse since the overwhelming majority of documents (99.something% last time I checked) uses pre-composed. Also AFAIK just about everyone just ignores that recommendation.
This is a classic "reality should adjust to the standard" type of thinking. Previous comments about that: https://news.ycombinator.com/item?id=36984331
I suppose "e ZWJ %" is a bit better than Private Use as it will appear as "e%" if you don't have font support, but the fundamental problem of "won't work unless you spend effort" remains. For a specific niche (math, language study, something else) that's okay, but for "casual" usage: not so much. "Ship font with the document" like PDF and webfonts do is an option, but also has downsides and won't work in a lot of contexts, and still requires extra effort from the author.
I'm not saying it's completely impossible, but certainly harder than it used to be, arguably much harder. I could coin a new word right here and now (although my imagination is failing me to provide a humorous example at this moment) and if people like it, it will see usage. In 1960s HN when we would have exchanged these things over written letters, and it would have been trivial to propose a "e with % on top" too, but now we need to resort to clunky phrases like this (even for typewriters you can manually amend things, if you really wanted to).
Or let me put it this way: something like ‽ would see very little chance of being added to Unicode if it was coined today. Granted, it doesn't see that much use, but I do encounter it in the wild on occasion and some people like it (I personally don't actually, but I don't want to prevent other people from using it).
None of this is Unicode's fault by the way, or at least not directly – this is a generic limitation of computers.
It shouldn't matter what's in the wild in documents. That's why we have normalization algorithms and normalization forms. Unicode was built for the ugly reality of backwards compatibility and that you can't control how people in the past wrote. These precomposed characters largely predate Unicode and were a problem before Unicode. Unicode won in part because it met other encodings where they were rather than where they wished they would be. It made sure that mappings from older encodings could be (mostly) one-to-one with respect to code points in the original. It didn't quite achieve that in some cases, but it did for, say, all of EBCDIC.
Unicode was never in the position to fix the past, they had to live with that.
> This is a classic "reality should adjust to the standard" type of thinking.
Not really. The Unicode standard suggests the normal/canonical forms and very well documented algorithms (including directly in source code in the Unicode committee-maintained/approved ICU libraries) to take everything seen in the wilds of reality and convert them to a normal form. It's not asking reality to adjust to the standard, it is asking developers to adjust to the algorithms for cleanly dealing with the ugly reality.
> Or let me put it this way: something like ‽ would see very little chance of being added to Unicode if it was coined today.
Posted to HN several times has been the well documented proposal process from start to finish (it succeeded) of getting common and somewhat less common power symbols encoded in Unicode. It's a committee process. It certainly takes committee time. But it isn't "impossible" to navigate and is certainly higher than "little chance" if you've got the gumption to document what you want to see encoded and push the proposal through the committee process.
Certainly the Unicode committee picked up a reputation for being hard to work with in the early oughts when the consortium was still fighting the internal battles over UCS-2 being "good enough" and had concerns about opening the "Astral Plane". Now that the astral plane is open and UTF-16 exists, the committee's attitude is considered to be much better, even if its reputation hasn't yet shifted from those bad old days.
> None of this is Unicode's fault by the way, or at least not directly – this is a generic limitation of computers.
Computers do anything we program them to do and in general people find a way regardless of the restrictions and creative limitations that get programmed. I've seen MS Paint drawn symbols embedded in Word documents because the author couldn't find the symbol they needed or it didn't quite exist. It's hard to use such creative problem solving in HN's text boxes, but that from some viewpoints is just as much a creative deficiency in HN's design. It's not an "inherent" problem to computers. When it is a problem they pay us software developers to fix it. (If we need to fix it by writing a proposal to a standards committee such as the Unicode Consortium, that is in our power and one of our rights as developers. Standards don't just bind in one-direction, they also form an agreement of cooperation in the other.)
This comes up in specifications that have a broad range of use cases; when I was involved in this my idea was to just spec things so that there's only one allowed form; you'll still need a small-ish table for this, but that's fine. But that's currently hard because for a few newer Latin-adjacent alphabets some letters cannot be represented without a combining character.
So then you have either the "accept that two things which seem visually similar are not identical" (meh) or "exclude embedded use cases" (meh).
I never really found a good way to unify these use cases. I've seen this come up a few times in various contexts over the years.
> Posted to HN several times has been the well documented proposal process from start to finish (it succeeded) of getting common and somewhat less common power symbols encoded in Unicode.
Would this work for an entirely new symbol I invent today? It's not really the Unicode people that are "difficult" here as such, they just ask for demonstrated usage, which is entirely reasonable, and that's hard to get (or: harder than it was before computers) especially for casual usage. I'm sure that if some country adopts/invents a new script today, as seems to be happening in West-Africa at in recent years, the Unicode people are more than amendable to work with that, but "I just like ‽" is a rather different type of thing.
Sure, they want demonstrated usage as inline in the flow of text as textual elements as opposed to purely iconography or design elements (because such things are outside of Unicode's remit, modulo some old Wingdings encoded for compatibility reasons and the fine line between emoji are expressive text and also emoji are useful for iconography in many cases). But at this point (again in contrast to the UCS-2/no-Astral-plane days) the committees don't seem to care how it was mocked up (do it on a chalkboard, do it in paint, do it in LaTeX drawing commands, whatever gets the point across) or how "casual" or infrequent the usage is, so long as you can state the case for "this is a text element" (not an icon!) used in living creative language expression. There's more "provenance" requirements for dead languages and they'll want some number of academic citations, but for living languages they've come to be flexible (no hard requirements) on the number of examples they need from the wild and where those are sourced from. Showing it in old classic documents/manuals/books, for instance, helps the case greatly, but the committees today no longer seem as limited to just what can be used to demonstrate usage. "I just like it" is obviously not a rock solid proposal/defense to bring to a committee (any committee, really), but that doesn't mean that is impossible for the committee to be swayed by someone making a strong enough "I just like it" case if they demonstrate well enough why they like it and how they use it and how they think other people will use it (and how those uses aren't just iconography/decorative elements but useful in the inline context of textual language).
Even "put the code points forming the composed character in descending numerical order" would be better than nothing. If it was there from the start.
However, the Unicode commitee is too busy adding new emojis to make their standard sane.
But, of course, unicode can't define that the standard will cover only the canonical forms, and couldn't do that since the start, as it needed backwards compatibility with various pre-unicode encodings which had mutually incompatible principles, so it needed support for both composed and decomposed versions of the same characters.
There' your problem right there. Plural formS. It's not canonical if there are more of it.
What on earth are you doing that it's leading to crashes? Are you not validating the result?
https://www.reddit.com/r/apple/comments/37e8c1/malicious_tex...
In fact, I don't know that there's any reason to believe normalization happens at all in the process of executing this.
Not just emojis, in general I believe Unicode has just said they're not going to add new pre-composed characters and that using combining characters is the Right Way™ to do things (well, the only way for newer scripts).
One of the downsides of writing down specifications is that they tend to attract people with Very Strong Opinions on the One And Only Right Way and will argue it to no end, and essentially "win" the argument just by sheer verbosity and persistence.
That's certainly what I've seen happen in a few cases, and is what happens on e.g. Wikipedia as well at times.
But yeah, emojis is even worse. Something things can look rather different depending on which invisible variation selector is present. We've got tons and tons of unassigned codepoints and we need to resort to these tricks to save a few of them?
Firefighter is "(man|woman|person) + ZWJ + firetruck". Clever, I guess. Construction worker is "Construction worker (+ ZWJ + (male sign|female sign))?" (absence is gender-neutral). Why are there 2 systems to encode this? Sigh...
All of this is too clever by a mile.
[1]: HN will strip stuff, but try something like:
echo $'↔\ufe0f ↔\ufe0e'
But the technical implementation? Yeah, that could have gone a lot better IMHO.
One must also wonder if some things really had to be added in the first place, e.g. for people kissing it's:
(person|man|woman)(skin-tone)? ZWJ <heart> ZWJ <kissing lips> ZWJ (person|man|woman)(skin-tone)?
I don't really get why <heart> or <killing lips> or <kissing face> isn't enough. That's actually what most people seem to use anyway, because who finds it convenient to pick all the correct genders and skin tones from the UI for both people?
Oh. So that's why HN discussion always looks sane. They strip the pollution.
Less than that since a default skin color can be set in most apps. I'm sure setting a gender will come soon so the entire first part of that emoji can be auto-guessed. Then its just showing the other options in the UI. Really all of this is UI design as even with the 98 combinations you can still display it as 4/5 options you drill down.
> who finds it convenient to pick all the correct genders and skin tones from the UI for both people?
I just checked and searching "kissing" in my iOS emoji keyboard inside Messenger showed just 4 of the emoji's your describing - defaulting both skin tones to my settings and then the four M/F pair ups. Plus some non-related kissing emojis like the cat kissing.
But that's kind of wrong, no? The entire point is that you can choose both sides individually. What if you set it to black and want to kiss some white bloke?
If anything that only underscores my point that it's too complex and that no one is using them (certainly not as intended anyway).
In the Windows 11 emoji picker it works like this:
1. Search "kissing". See two generic yellow people kissing. Notice a blue dot in the bottom right corner.
2. Clicking the emoji brings up previously used versions of the kissing emoji, with a + button.
3. Clicking + brings up a dialog like I described previously. Two generic figures at the top, then a row of skin tones.
4. You can click on each generic person and choose a gender, then select a skin tone. You can do this for each person in the group.
5. Click done. This emoji is now in your default emoji list and you won't need to recreate it again.
Also, the % should measure people, not languages, that would greatly decrease the imaginary 99%
I didn't know that unicode changes the definition of grapheme in backwards incompatible fashion annually, so software which works by grapheme count is probably inconsistent with other software using a different version of the standard anyway.
I'm therefore going to continue counting bytes. And comparing by memcmp. If the bytes look like unicode to some reader, fine. Opaque string as far as my software is concerned.
e.g., a TUI with columns will have to truncate "long" strings in each column, and that truncation and column-separator arrangement really should be grapheme aware.
e.g., a string search (for a name, let's say) should find Noël regardless of whether the user input ë via composing characters or the pre-composed version.
This is EXACTLY why Rust's standard library is blind to graphemes. Support for the case where your company requires a specially certified toolchain that lags five years behind Rust upstream is an explicit goal that they address by breaking the stuff that changes quickly out into minimal crates that can be audited, updated at a quicker pace without requiring toolchain updates, and which have the option to continue to support older compiler versions indefinitely.
...and learn about the TR39 Skeleton Algorithm for Unicode Confusables. Far too few people writing spam-handling code know about that thing.
(Basically, it generates matching keys from arbitrary strings so that visually similar characters compare identical, so those Disqus/Facebook/etc. spam messages promoting things like BITCO1N pump-and-dumps or using esoteric Unicode characters to advertise work-from-home scams will be wasting their time trying to disguise their words.)
...and since it's based on a tabular plaintext definition file, you can write a simple parser and algorithm to work it in reverse and generate sample spam exploiting that approach if you want.
https://www.unicode.org/Public/security/latest/confusables.t...
> and CD-ROM!
I think you mean Microsoft Windows's Joliet extensions to ISO9660 which, by the way, use UCS-2, not UTF-16. (Try generating an ISO on Linux (eg. using K3b) with the Joliet option enabled and watch as filenames with emoji outside the Basic Multilingual Plane cause the process to fail.)
The base ISO9660 filesystem uses bytewise-encoded filenames.
I normalize strings in searches not because of bad intents but because for all user related purposes "Comunicações" and "Comunicações" are the same, their different encodings being more of an accident.
I was just pointing out that Unicode itself has various forms of normalization and normalization-adjacent functionality that people are far too unaware of.
Problem solved!
But any more saturation and it would go all mustard-electric on me.
That's an interesting observation on variation of saturation response. Feels like useful knowledge for ... web site designers. Or any color crafter.
It's black text on black background (I'm on mobile Firefox on Android).
Future _improvement_ idea: the mouse cursors are shared, so the light switch should be, too! Let me play with the light with everyone
Deleting that makes the while thing a lot less stressful.
Found this! good to know about. https://pypi.org/project/grapheme/ "A Python package for working with user perceived characters. "
(apparently the article talks about this however the blog post is largely unreadable due to dozens of animated arrow pointers jumping all over the screen)
Oddly, even though the Word UI indicated it was Arial, exporting to a PDF and inspecting that revealed that Segoe UI Symbol was being used.
As I've noted in the past, "If typography was easy, Microsoft Word wouldn't be the foetid mess which it is."
I wish that was the only thing wrong with that page
“And a couple of important consequences:
- You CAN’T determine the length of the string by counting bytes.
- You CAN’T randomly jump into the middle of the string and start reading.
- You CAN’T get a substring by cutting at arbitrary byte offsets. You might cut off part of the character.”
One of the things I had to get used to when learning the programming language Janet is that strings are just plain byte sequences, unaware of any encoding. So when I call `length` on a string of one character that is represented by 2 bytes in UTF-8 (e.g. `ä`), the function returns 2 instead of 1. Similar issues occur when trying to take a substring, as mentioned by the author.
As much as I love the approach Janet took here (it feels clean and simple and works well with their built-in PEGs), it is a bit annoying to work with outside of the ASCII range. Fortunately, there are libraries that can deal with this issue (e.g. https://github.com/andrewchambers/janet-utf8), but I wish they would support conversion to/from UTF-8 out of the box, since I generally like Janet very much.
One interesting thing I learned from the article is that the first byte can always be determined from its prefix. I always wondered how you would recognize/separate a unicode character in a Janet string since it may have 1-4 bytes length, but I guess this is the answer.
There's no random access in Unicode. It's a stateful system that requires linear scan.
I believe there actually are topics that every software developer ought to know something about, but this isn't one of them. My list would be things more like, the difference between a constant-time algorithm and a quadratic-time one.
However, as soon as you want to look into the text contents of the objects of that object store and handle parts of it, even in the very simplest way (e.g. checking whether the stored object contains some character, or whether two text messages stored in that object store are the same) then you can't treat them as bytes anymore, and all the concerns listed in this article suddenly become relevant for your team.
$ perl -wle 'use utf8; print length("");' 1
Without use utf8: $ perl -wle 'print length("");' 3
It's funny: after Perl fell out of favour, is when it got all its best stuff. It's still my preferred language for just about everything.
print("...".count)
// => 1
% erl +pc unicode
> string:length("...").
1
> UPD: Erlang/Elixir seem to be doing the right thing, too.
https://github.com/tonsky/tonsky.me/commit/5fbbb373025be3758...
Does anyone know how to write a function (preferably in swift) to remove emoji? This is surprisingly hard (if the string can be any language, like English or Chinese).
There’s been multiple attempts on Stackoverflow but they’re all missing some of them, as Unicode is so complex.
`String("text EMOJI 0123 漢字".unicodeScalars.filter({ !$0.properties.isEmojiPresentation }))`
(I've had to substitute EMOJI for a smiley face, because HN is bad at text encoding.)
This is nonsense, Bulgaria has been using the Cyrillic alphabet sinse its creation in … Bulgaria!
What you’ve shown is two different fonts, and both renderings are perfectly fine in Bulgaria.
Read up more about it on wikipedia: https://en.wikipedia.org/wiki/Bulgarian_alphabet
A quick war-story on this: We had a system which was taking web-user input for human names, and then some of it had to be sanitized for a crappy third-party system. However some of the names were getting mangled in unexpected ways.
One of the (multiple) issues was that we were sometimes entirely dropping accented characters even when a good alternative existed. This occurred when we were getting "é" (U+00E9) instead of "é" (U+0065 U+0301), a regular letter E plus a special accent modifier. By forcing the second form (D normalization) we were able to strip "just the accents" and avoid excessively-wrong names.
Going further with K+D normalization, weird stuff like "⑧" (letter 8 in a circle) becomes a regular number 8.
(Among other things, it points out that doing that to non-Latin text is liable to change pronunciations and meanings in other languages. For example, some languages use diacritics for voiced/unvoiced indication where your "normalization" could do things like "tick→dick" or "did→tit".)
(Did you ever notice that? B/P, D/T, V/F, G/K, J/CH, and Z/S form voiced/unvoiced pairs that could have been indicated with a single letter and a diacritic. Same mouth behaviour. It's just a question of whether you engage your vocal cords.)
> The only modern language that gets it right is Swift:
julia> using Unicode
# for some reason HN doesn't allow emoji
julia> graphemes(" ")
length-1 GraphemeIterator{String} for " "
help?> graphemes
search: graphemes
graphemes(s::AbstractString) -> GraphemeIterator
Return an iterator over substrings of s that correspond to the extended graphemes in the string, as defined by Unicode UAX #29. (Roughly, these are what users would perceive as single characters, even though they may contain more than one codepoint; for example a letter combined
with an accent mark is a single grapheme.)For example indexing and length of the string are done by codeunit. [1]
On the other hand Rakus Str type does behave similarly to Swifts: indexing, length and iteration by grapheme; view methods for specific encodings. [2]
[1]: https://docs.julialang.org/en/v1/base/strings/ [2]: https://docs.raku.org/type/Str#routine_chars
Just a nitpick...
Once more, as it is typical on HN, web programming is confused with the entire universe of software development.
There are plenty of software realms where ASCII not only is enough, but it actually MUST be enough.
Not being able to support non-latin scripts sounds more like a limitation than a feature to me, although of course in many contexts it’s not in any individual organizations power to overcome it.
"Web" programmers can care all they want about Unicode, but if the backend people didn't deal properly with text encoding, then something will break no matter what.
> There are plenty of software realms where ASCII not only is enough, but it actually MUST be enough.
Name one.
You are right. It's not a frontend/backend issue. It's a "for human" vs "not for human" issues. Personal names must be treated in an international-friendly manner.
>> There are plenty of software realms where ASCII not only is enough, but it actually MUST be enough. > > Name one
Joel himself described an example:
> It would be convenient if you could put the Content-Type of the HTML file right in the HTML file itself, using some kind of special tag. Of course this drove purists crazy… how can you read the HTML file until you know what encoding it’s in?! Luckily, almost every encoding in common use does the same thing with characters between 32 and 127, so you can always get this far on the HTML page without starting to use funny letters:
The content of a webpage is required to be expressed in every supported language, but the HTTP protocol must not. And it would make no sense at all to add internationalization to intra-machines protocol, where ASCII is enough and has been enough for decades.
And if someone complains that ASCII only supports English, well... suck it up! I'm Italian and work in French, still I hate when a colleague sneaks in a comment not in English. Professional software development happens in English.
I guess no URLs with funny characters then. "GET /profile/renée" => 500 error, woohoo.
> And if someone complains that ASCII only supports English, well... suck it up! I'm Italian and work in French, still I hate when a colleague sneaks in a comment not in English. Professional software development happens in English.
Get over yourself, a lot of professional development happens in languages other than English.
That's not really a slam dunk. Lots of sites don't let you have your name in the URL at all, and the average person's experience is that their name would be taken by someone else before they signed up.
In the metaphor, it's someone telling you your aim is off.
UTF8 encoded diacritics work just fine in C++.
That's all that's needed for a backend language.
The backend does not need to understand, or even acknowledge the existence, of grapheme clusters. Because the frontend is already having to understand all of this, it should be normalising any multi-codepoint ambiguous cluster anyway.
Not as measured by clusters, no.
> search for one string in a database of other strings?
Hence I said "normalisation". The frontend already has to do all the unicode twiddling, it may as well normalise the input too.
We absolutely do not "need" to know about Unicode, outside of interest about other realms.
"The problem is, in Unicode, some graphemes are encoded with multiple code points!"
"An Extended Grapheme Cluster is a sequence of one or more Unicode code points that must be treatead as a single, unbreakable character."
"Starting roughly in 2014, Unicode has been releasing a major revision of their standard every year."
"Å" === "Å" "Å" === "Å" "Å" === "Å" What do you get? False? You should get false, and it’s not a mistake.
"That’s why we need normalization."
"Unicode is locale-dependent"
The article forgot one: characters that switch presentation to right-to-left.
It occurs to me that a canonical semantic representation of all known (extracted) language concepts would be useful too.
Now that we have multi-language LLM's it would be an interesting challenge to create/design a canonical representation for a minimum number of base concepts, their relations and orthogonal "voice" modifiers, extracted from the latent representations of an LLM across a whole training set, over all training languages.
While the best LLMs still have complex reasoning issues, their understanding of concepts and voice at the sentence level is highly intuitive and accurate. So the design process could be automated.
The result would be a human language agnostic, cross-culture concept inclusive, regularized & normalized (relatively speaking) semantic language. Call it SEMANTICODE.
We need to get this right, using one standard LLM lineage, before the Unicode people create a super standard that spans 150 different LLM's and 150 different latent spaces! :O
Stability between updates would be guaranteed by including SEMANTICODE as a non-human language in training of future LLM's. Perhaps including a (highly) pre-normalized semantic artificial language would dramatically speed up and reduce the parameter count needed for future multi-language training?*
Then LLMs could use SEMANTICODE talk to each other more reliably, efficiently, and with greater concept specificity than any of our single languages.
I stopped reading your article because of your little websocket experiment.
This is the most interesting comparison article I have seen in years about Unicode processing in C++: https://thephd.dev/the-c-c++-rust-string-text-encoding-api-l...
The author is also the lead on an open source C++ Unicode library called ztd.txt: https://github.com/soasis/text
That coloration tools shows the moving mouse coursers of other participants even if they aren't needed/wanted is already pretty bad, why bring it to a website?
through it also wasn't meant to be constructive critique
It's like saying that we don't need to tokenize, because you never want to deal with tokens anyway, but phrase structures!
Mmkay, whatever ...
>It is important to recognize that what the user thinks of as a “character”—a basic unit of a writing system for a language—may not be just a single Unicode code point. Instead, that basic unit may be made up of multiple Unicode code points. To avoid ambiguity with the computer use of the term character, this is called a user-perceived character. For example, “G” + grave-accent is a user-perceived character: users think of it as a single character, yet is actually represented by two Unicode code points. These user-perceived characters are approximated by what is called a grapheme cluster, which can be determined programmatically.
It's complexity is a relief compared to where we've been. It's definitely not simple, but it will forever be far simpler than what our grandmothers had to work with if they were writing international software.
TL;DR: Unicode is complicated because some non-Latin writing systems are complicated and those non-Latin writing systems account for over a quarter of the world's population. (They're either majority or present in India, Indonesia, Pakistan, Bangladesh, the Philippines, etc.)
From everything i've read or heard about unicode, "determined programmatically" is false?
These are part of language and should not be an optional proprietary add on that can be skipped or deleted from text. We've been using the two "formats" to convey important information since the sixteenth century!!!
It boggles my mind that we can give flesh tone to emojis, yet not mark a word as bold or italic. It makes zero sense. Especially how easy it would be to implement. It would work exactly the same way: Letters following the mark would be formatted as bold or italic until a space character or equivalent.
In what twisted reality did someone think this a good idea?
Doesn't it go against the whole premise of everyone in the world agreeing on how to represent a meaningful unit of text?
"What’s sad for us is that the rules defining grapheme clusters change every year as well. What is considered a sequence of two or three separate code points today might become a grapheme cluster tomorrow! There’s no way to know! Or prepare!"
"Even worse, different versions of your own app might be running on different Unicode standards and report different string lengths!"
Skipping over UTF-32-BE and UTF-32-LE there...
(I mean, it might not be an issue if it's just being used as an internal representation, but still)
Wow, what abominable mix of decimal and hexadecimal.
doh.
So the author is using hex for the last four digits and decimal for the remaining ones.
This is a nice example of the kind of thing we need to think about when defining a measure of length for Unicode strings.
The more interesting question is whether the Unicode rules actually give that answer.
EDIT: Just checked it using the first online tool [1] that came up and it indeed says four. So all is good.
[1] https://onlinetools.com/unicode/extract-unicode-graphemes
In Javascript, you can get the same information through Intl.Segmenter, segments by grapheme cluster by default.
This isn't quite Unicode's fault, as the alternative would be to have two codepoints each for `i` and `I`, one pair for the Latin versions and one for the Turkish versions, and that would be very annoying too.
Whereas the Russian/Bulgarian situation is different. There used to be language tags in Unicode for that, but IIRC they got deprecated, and maybe they'll have to get undeprecated.
Most languages, when comparing strings, skip the normalization and just compare string bytes as is (or, if the string is interned, compare just the pointer)
Fortunately, it seems like it's marked as deprecated: https://en.wikipedia.org/wiki/Angstrom#Symbol
Frequently, the answer to this is "some obscure character set had this as a distinct symbol." In this case, blame the Japanese: https://en.wikipedia.org/wiki/JIS_X_0208
Which is why there's an 'mm' and 'cm' and other random symbols: https://www.compart.com/en/unicode/block/U+3300
Now they look different on every platform and many subtleties are lost in translation.
It's probably a good thing, though.
Apple did a fairly good job with unicode string handling starting in Cocoa and Objective-C, by providing methods to get the number of code points and/or bytes:
https://stackoverflow.com/questions/15582267/cfstring-count-...
I feel that this support of both character count and buffer size in bytes is probably the way to go. But Python 3 went wrong by trying to abstract it away with encodings that have unintuitive pitfalls that broke compatibility with Python 2:
https://blog.feabhas.com/2019/02/python-3-unicode-and-byte-s...
There's also the normalization issue. Apple goofed (IMHO) when they used NFD in HFS+ filenames while everyone else went with NFC, but fixed that in APFS:
https://unicode.org/faq/normalization.html
https://medium.com/@sthadewald/the-utf-8-hell-of-mac-osx-fee...
No. In my browser the first A has the ring glued to it, and the second has a little gap.
Well, there is a new fact that I learned and immediately hated.
The fuck were authors thinking...
I am now firmly convinced people developing unicode hate developers. I suspected it before just due to how messy it was (same character having different encodings ? Really ? Fuck you), but this cements it.
But making it clear which glyph to use is also a key feature!
Or at least they have a vicious indifference to us. Unicode is a nightmare.
Hmm? I thought some code points combine to create a character. Even accented latin ones can be like that.
Also we need to agree on what is a character.
Indeed. I used to think I knew what a character was until Unicode came around. Now I genuinely don't know with any real certainty.
Look, I would love to be able to read/write Sanskrit, Arabic, Chinese, Japanese etc and share those content and have everyone render and see the same thing. The problem is that I feel like most of these are:
1. a kind of an open problem 2. very subjective 3. very, very subjective as what you is mostly dictated by the implementation (fonts)
For example, why does a gun emoji looks like water gun? Why is the skull-and-crossbones symbol looks so benign. In fact, it is often used as a meme (see deadass :skull:) Why is the basmala a single "character"?
In my opinion, people should just learn how to use kaomoji. Granted, kaomojis rely on a lot more than the Latin characters but it is at least artful, skillfull and a natural extension of the "actual" languages.
> inb4 languages evolves
Yes, but it mostly happens naturally. I feel like what happens today mostly happens at the whim of a few passionate people in the standard.
I don't want to support emoji either (and, I don't want emoji on my computer), although in some cases, if it is really necessary to be supported, they could be implemented just as text characters instead of as colourful emoji, anyways.
For many purposes (e.g. computer codes) ASCII is good enough (and actually even can be better since it avoids the security problems of using Unicode). (Sometimes, character sets other than ASCII can be used, e.g. APL character set for APL programming.)
> Look, I would love to be able to read/write Sanskrit, Arabic, Chinese, Japanese etc
I also would, but Unicode is bad enough that I would use other ways of doing such a thing when possible (even writing my own programs, etc). (If a program insists on Unicode, I might just use ASCII only anyways, or write my own program)
Not everyone necessarily need to see the same thing (if it is a text, rather than pictures of the text), although, the suitable character sets for that language which can be in use (and with fix pitch if necessary, etc), to auto select a suitable fonts for your computer by the reader's preference.
So, I prefer to support all languages (where applicable; sometimes it isn't), without using Unicode.
The standard library provided by languages like C, C++ is a library. Features like character strings are present and it's a totally reasonable expectation for the length to give you the cluster count.
If you want cluster count, call a different function.
1.1 million?
The only modern language ((he knows of)) that gets it right to be precise. Ruby also gets it right.
``` [1] pry(main)> "".size => 1 ```
Better yet, there shouldn't be a function called length.
Elixir too:
Interactive Elixir (1.15.4) - press Ctrl+C to exit (type h() ENTER for help)
iex(1)> String.length "ẇ͓̞͒͟͡ǫ̠̠̉̏͠͡ͅr̬̺͚̍͛̔͒͢d̠͎̗̳͇͆̋̊͂͐"
4EDIT: I've never been downvoted for asking a question before. Weird, but okay.
Why is Tengwar still not in Uniclde officially? What's the problem with it?
Copyright is not technically violated simply by encoding the characters into a plane such as one of Unicode's, that's an easy open and shut fair use, but Unicode principals have stated they don't want to pass on the copyright burden to font authors either, which would be sued if they tried to paint some of those characters. (Why encode something that fonts aren't allowed to produce?) That should also be fair use, but the law is complicated and copyright still so often today leans in favor of the Estates and major Corporations rather than fair use and the public commons.
(ETA: I'm hugely in favor that "conlang", constructed language, scripts such as these should be encoded by Unicode. I wish someday we fix the copyright problems of them.)
I must have missed something. All of those symbols are present in Extended ASCII (i.e. 8-bit).
/^[A-Za-z\-' ]+$/
What every software developer must know about IPv6 in 2023 (still no excuses!).
The .graphemes() method in Rust's unicode-segmentation crate takes an is_extended boolean as an argument and, if you set it to false, you're iterating legacy grapheme clusters.
TL;DR: They had a transient glitch in their network switch and, because Windows uses UTF-16 when sending remote event logs over the wire, whenever it dropped a single byte, it had the effect of swapping the endianness of the messages, resulting in scary Chinese text in the logs.
You could get the same effect by naively applying byte-wise processing to UTF-16 or UTF-32, or having an off-by-one error.
UTF-8 is self-synchronizing so one-byte errors like that only lose you one character, rather than corrupting the entire stream going forward.
Unicode just took it upon themselves to reliably round-trip legacy text... thus the precomposed forms.
Most of the other complexity and technical debt is in the writing systems themselves.
Rather not. It takes an incredible amount of work to get it right. Just stick to ICU.
But there is not a code for each flag. Instead there is a code for each ASCII character. A flag sequence is formed from U+1F3F4 (Black Flag), followed by at least two tags that form a country/region code, and then U+E007F (End tag).
So, yes this is weird, because the emoji is dependent on the decoder. It was made this way to keep Unicode independent of geopolitics.
Read more: <https://en.wikipedia.org/wiki/Tags_(Unicode_block)>
As for the mouse cursors, I don't think they qualify as personal information under the GDPR, but IANAL.
Regardless, I don't think it matters since the author is not in the EU.
This content must be really awesome for someone to go through the trouble of interacting with such a site.
I'm sure the article is good but the title is nonsense.