Hacker News Reader

Comment by boolemancer | Hacker News Reader

> P.S. Having multiple Unicode values that exhibit identically when displayed are a huge veer-into-the-ditch mistake. I.e. the notion that code points should have semantic value is simply wrong.

Should a cyrillic `а` and a latin `a` have the same code point?

If they did then there's no consistent way to group those two alphabets, one or the other would end up with letters outside it's main grouping.

And what happens if the shapes of those two characters divege over time? Do you have a breaking change to introduce a new code point for one of the languages, or do you just make it impossible to have a font that can be used for both latin and cyrillic at the same time?

This isn't just a theoretical problem, by the way. There are characters in Chinese and Japanese that share a code point but the shapes of the characters aren't the same in the two languages.

View on HN

> Should a cyrillic `а` and a latin `a` have the same code point?

Yes. Consider a book. Can you tell if it's a cyrillic or a latin `a`? Of course you can, because of the context. Unicode is about visible text, having hidden semantic meaning makes it something else.

Besides, 'a' can have all kinds of semantic meanings - all depending on the context in which they are used. There is no way to encode all this into Unicode.

> If they did then there's no consistent way to group those two alphabets

Doesn't matter.

> And what happens if the shapes of those two characters divege over time?

If it actually looks different, then it becomes a different code point.

> There are characters in Chinese and Japanese that share a code point but the shapes of the characters aren't the same in the two languages

More evidence that the Unicode committee lost its way.

Think of it this way. Printing Unicode text on a piece of paper, and then OCRing it back into Unicode, should be a lossless operation. Or another way - anyone should be able to tell what the code point value is by looking at the visual representation of it.

How about an l and an I? Or a closed "a" and one with the little handle? A zero with a stroke, a zero without a stroke, and an O?

I can see where you're coming from, but deduplicating every glyph from every culture based on which do or don't generally look the same when printed sounds like a tall order. And if all you want to record is the shape, you can use a PNG with OCR and bypass Unicode entirely.

> How about an l and an I? Or a closed "a" and one with the little handle? A zero with a stroke, a zero without a stroke, and an O?

Those are font differences, not character differences. (Unicode has also failed by adding in some fonts. The nuttiness never ends.)

But just as the “a” in alphabet and the «а» in азбука are only distinguishable by context, so too the “O” in SOS and the 0 in 90210 are often only distinguishable by context. I don’t see how you propose to determine which glyphs should be the same and which should be different in your system.

> so too the “O” in SOS and the 0 in 90210 are often only distinguishable by context

Only if it's a bad font.

How does your scheme handle Turkish "Dotless I"?

Should there be different codepoints for serif and sans-serif versions of the same letter? After all, in some typefaces, "uppercase I" and "lowercase L" look just about the same.

I think you mean fonts. No, Unicode should not encode fonts.

Nor should it encode italic, boldface, underline, line out, reverse video, point size, superscript, subscript, or colored. Those are all style attributes, best applied with a style sheet, not a code point.

> superscript, subscript

Very strongly disagree. Copy-pasting 10<sup>6</sup> turns into 106 rather than 10⁶.

> If it actually looks different, then it becomes a different code point.

Which language gets the original code point and which one needs to change?

And what happens to all of the existing text written with the original code point?

Obviously we are stuck with Unicode as-is now, warts and all. But that doesn't mean that the original design was the right one.

Reply on news.ycombinator.com