Hacker News Reader

Comment by pif | Hacker News Reader

> An extended grapheme cluster is largely defined on "this visually displays as a single unit", which isn't necessarily correct for things like "display size in a monospace font" or "thing that gets deleted when you hit backspace."

I'm sorry, but I fail to see how "This visually displays as a single unit" could ever differ from "Display size in a monospace font" or "Thing that gets deleted when you hit backspace".

View on HN

A couple of cases I'm aware of...

* Coding ligatures often display as a single glyph (maybe occupying a single-width character space, or maybe spread out over multiple spaces), but are composed of multiple glyphs. The ligature may "look" like a single character for purposes of selection and cursoring, but it can act like multiple characters when subject to backspacing.

* Similarly, I've seen keyboard interfaces for various languages (e.g., Hindi) where standard grapheme cluster rules bind together a group of code points, but the grapheme cluster was composed from multiple key presses (which typically add one code point each to the cluster). And in some such interfaces I've seen, the cluster can be decomposed by an equal number of backspace presses. I don't have a good sense of how much a monospaced Hindi font makes sense, but it's definitely a case where a "character" doesn't always act "character-like".

I've always felt ligatures that condense two or more glyphs into something that takes up the space of only one in a monospace font are going beyond what a font should handle and into the realm of what an editor should do. I have several such visual substitutions set up in my .emacs but I don't use fonts that do them on their own.

What about ligatures that make ASCII characters display differently when in proximity, but still use the same number of columns?

For example, when == is written, connect them to be a 2 column wide = instead.

Or when === is written, display a three column equals sign, but it's three bars instead of two.

> Display size in a monospace font

Some clusters are going to be multiple characters wide.

> thing that gets deleted when you hit backspace

Some clusters are meant to be composted of multiple keystrokes and a natural editing experience would allow users to delete the last stroke.

Look into how Korean works.

See, e.g., https://github.com/xi-editor/xi-editor/issues/655 for why backspace isn't the same as extended grapheme cluster.

As for "display size in monospace font", emojis and CJK characters are usually two units wide, not one (although, to be honest, there's a fair amount of bugs in the Unicode properties that define this).

Here’s a good example of the test cases used for backspaces in Android[1]. It’s definitely more involved than just deleting a grapheme cluster.

[1] https://android.googlesource.com/platform/frameworks/base/+/...

Here is a full article of such examples: https://manishearth.github.io/blog/2017/01/14/stop-ascribing...

Discussion on HN: https://news.ycombinator.com/item?id=31858311

If you type "a", combine it with "´", then change your mind and hit backspace, you probably want to end up with "a" even through "á" was a thing "visually displayed as a single unit".

But then if I type "á" directly (through, say, a mobile keyboard) and hit backspace, I'd get "a", which doesn't seem terrible but does feel a little off.

Seems like the right answer for codepoints vs graphemes, unfortunately, is dependent on the context.

I expect to delete the character "á". And I prefer consistency too so I expect "œ" and "<emoji>" and "<emoji>" to be deleted as one unit.

edit: emojis are filtered by HN

Even the emoji's that you create by combining multiple emojis? Type one emoji, then a second, it merges into one. What happens when you backspace?

I expect the full emoji to be deleted. Because it merged into one visual entity.

I guess it's more ambiguous for some languages that can have long ligatures though.

I'd say delete the whole with backspace, but only the last if you undo.

Most European keyboard layouts have it the other way around: first press a "dead key" for the diacritic mark and then the letter to apply it to.

Where some layouts may require this method for some characters, another keyboard layout may have the same character on a dedicated key.

The program receives the combined character as one unit, and does not need to be aware of different keyboard layouts.

> Most European keyboard layouts have it the other way around: first press a "dead key" for the diacritic mark and then the letter to apply it to.

Which ones? At least the French and German ones don’t work like that: there is no composing, just separate keys for all the characters with diacritics that appear in the language.

Which French layout would that be? I've never seen a French keyboard where this is true. French is my native language. On layouts I'm familiar with, some accented letters have separate keys like é, but not all, the others are made by composing an accent key with a letter.

You're right, sorry. I had forgotten about the ^ and ¨ keys.

Slovak or Czech for example.

On the German Layout the backtick (next to the 1 key) is a dead key.

Whichbis the thing that finally pushed me over the edge and switch to the US layout. backticks are something I use all the time.

Danish keyboards also require you to press '¨' first and then 'o' to produce 'ö'.

Do the danes not have the mechanism that is found on Finnish keyboard layouts, where pressing AltGr+Ö yields Ø and AltGr+Ä yields Æ, except in reverse?

Those mappings are not universal. They are present under Linux but not on MS-Windows. I don't know about Mac, but the layout has in the past been slightly different there from Windows also.

Interesting, it's been like a decade since I last used windows, but I had to go and check what layouts are available, since I remember having these combinations on my layout. Apparently those combinations are provided by windows in the "Finnish and Sami" layout, which provides a number of extra letters (not just ones used by the Sami languages) through AltGr+letter combinations. I must have selected that as my layout at some point while I was still using windows, possibly for the purpose of getting easier access to letters like ÆØÕ, and just forgotten it after some time.

They are there on mac too, use option-Ä to get the Æ or the other way around. What's more, it has worked like that since system 7.x times or so, it's just a good idea.

For me that doesn't work on Windows. Those key combinations doesn't seem to do anything.

I had to change settings on Windows to get access to a mode. I could then enable that mode to be able to readily type Spanish correctly. The mode uses the key combinations as described.

But do you really use ö much over ø?

No, but I do once in a while (very rarely) write a little in german that might use that character.

Nitpicking but most french keyboards have both ready-made keys for "é" and the few other commonly use keys and composing: hitting either '¨' or '^'. For example hitting '¨' then 'e' produces "ë".

You are right, thanks.

The AZERTY layout is nothing if not inconsistent.

For this specific example, it is actually quite pragmatic. "é" being used many orders of magnitude more often than "ë" in French, it makes sense for it to have its own key.

Also, French has no other character that takes an acute accent. For the same reason, ç isn't typed with a dead key on French AZERTY.

The nordic layout(s) offer such a mechanism to allow people to type in letters that you'll find in various other European languages, even though the extra letters used in the languages themselves (ÅÄÖÆØ) are present as their own keys. Interestingly, the Swedish layout has no dedicated é key, although é occurs in some Swedish words.

In Swedish, Å, Ä, and Ö are actual letters of the alphabet, while é is used in foreign words. Like the English dieresis (e.g. in coöperate) is essentially unknown in the US and only occasionally used in England, so doesn't give rise to characters with dieresis on the keyboard.

é is used commonly in names and some words that don't feel foreign. For example the word for idea is written idé. Seems like it's an old loan from greek.

The accent gives away that this is in fact a loan from French

I guess it's time to learn to use some real swedish words then and not the foreign ones. Bye idé, hello hugskott. (Hug/håg = mind, skott = shot)

Danish is one.

Press the key to the left of 1 (not the numpad) or the right of the Eszett (sharp S) on the German QWERTZ keyboard and you probably hit a dead key. There are dedicated keys for the German umlauts and Eszett but these are for French accents in loan words: â, á and à, e.g. as in Café.

It's worth mentioning of course that there are no-dead-keys variants of the keyboard layout but this has been pretty much the norm on Windows since the 1990s I think.

> first press a "dead key" for the diacritic mark and then the letter to apply it to.

That being exactly the way “floating diacritics” in ISO 2022 (or properly one of its Latin encodings, T.51 = ISO 6937) work, amusingly. I wonder which came first. (Yes, I know that a<BS>` came first, the ASCII spec even says that this should give you an accented character IIRC. Or perhaps it was one of the other “don’t call it ASCII” specs—ISO 646? IA5?..)

As a European, no I don't. á isn't used in my language, but my layout offers it via a dead-key-then-base-letter mechanism, and it is correctly treated as one unit when pressing backspace, anything else would feel incorrect. It would be even worse if such a thing happened for the letters that my layout offers individual buttons for (ÅÄÖ). Some languages do treat these as letters with attached modifiers, but many, including mine, treat them as indivisible letters that just happen to look similar to some others for historical reasons, and to treat them as combinations of base letters and diacritics would be completely incorrect, even if you typed them in using the dead-key-then-base-letter mechanism for some reason.

In terminals there is a distinction between single-width and double-width characters (east-asian characters, in particular). E.g. the three characters

    A美C

would take up the width of four ASCII monospace characters, the “美” being double-width.

Similarly, for composed characters like say the ligature “ﬀ”, you may want to backspace as if it was two “f”s (which logically it is, and decomposes to in NFKD normalization).

Unicode even has distinct full width and half width variants of Japanese katakana - where ‘full width’ is (in theory) as wide as two Latin characters.

   Latin:      Katakana
   Full width: カタカナ
   Half width: ｶﾀｶﾅ

(How that fixed width text looks in a web browser is anyone’s guess though. On iOS none of the Japanese kana stay on the fixed grid.)

The background here is that both are contained in the Japanese Shift JIS character set, and Unicode provides roundtrip compatibility. And they are in Shift JIS because the half-width katakana were in the 8-bit JIS character set [0] used with text-mode displays where all characters have the same width. To preserve screen layout, these later had to be distinguished from full-width katakana.

[0] https://en.wikipedia.org/wiki/JIS_X_0201

﷽

𒐫

𒈙

꧅

If I have a text with niqqudim I am going to want to think of the niqqudim differently when editing despite the fact they are entwined with the consonants.

characters that alter their appearance to be one or more display units depending on the characters that are next to it (before and after). that would be a very crude example, but these types of characters appear all the time in human language

Reply on news.ycombinator.com