Unicode 15.0 Slide Show | Hacker News Reader

Unicode 15.0 Slide Show

babelstone.co.uk

114 pointsoptimalsolver71 comments

Wikipedia has the “no original research” rule. Unicode really should have had a similar “no original designs” rule.

Too late now — it’s become an annually refreshed collection of fun fashionable clip art instead of an impartial repository of humankind’s symbols.

Unicode's stuck to that principle better than Wikipedia has stuck to its principle.

Here's the blog post for the Unicode 15 release: https://blog.unicode.org/2022/09/announcing-unicode-standard...

A tiny portion consists of "fun fashionable clip art." The vast majority of the changes are new scripts and symbols that allow people to correctly input existing texts.

Ah, but the "Zoomer mind rot clip art YOLO YEET SOY LATTE" is what I can use to get all pissy and stuff, so it must be what the Unicode people focus on!

I think there's plenty of evidence that people use pictograms. Just look at the millions of discord emoji and chat stickers. In that respect, they're showing remarkable constraint in only adding twenty.

tbh my bigger worry is simply around implementation scope and complexity - kinda like browsers, Unicode and text rendering is perpetually getting harder to create new competitors / complete fonts / etc. Emoji are rather small in number and simple to implement though, if somewhat costly to include all those images / complex svgs.

You can hire a ghostwriter to write a biography about yourself and pay a publishing house for a limited edition print run and voila, you can have your own wikipedia page based on an authorized source.

Wikipedia's view of what's 'original research' has alway been a bit murky, on top of that. Are the primary documents that historians rely on acceptable as sources, or only if they've been synthesized by an 'accredited historian' (whatever that is) in book or research journal form? If publishing original research is wrong, why publish a journalist's original research which is incorporated into a newspaper article? What about original research published as a blog post, is that now an acceptable secondary source?

> You can hire a ghostwriter to write a biography about yourself and pay a publishing house for a limited edition print run and voila, you can have your own wikipedia page based on an authorized source

It isn’t as easy as you make it sound - to establish notability, they don’t accept self-published / vanity press sources - so you’d need to get your book published by a publisher with an established track record. That’s a lot harder - it is either something that money can’t buy, or at least you’d need a lot lot more money to buy it than self-publishing charges

Furthermore, one high quality reliable source is generally not considered enough for notability, they want multiple sources. If you get your biography published, and then get some journalists in established media outlets to publish articles about it, you’ll meet that hurdle too. But if you manage that, you are probably actually are notable, as opposed to just some random nobody trying to buy their way into Wikipedia

Unicode has completely separate rules for characters and emojis. Characters and scripts generally need solid documentation that they are real existing symbols. Emojis, on the other hand, are accepted largely on how likely they are to be used. My impression is that the Unicode committee would prefer to deal with scripts and characters, but they got stuck with emojis for historical reasons and that's what most people care about.

Refs: https://www.unicode.org/emoji/proposals.html http://www.unicode.org/pending/proposals.html

No, man. Digital communication should follow whatever history for inclusion that text before 1850 did. Write a `:P` (you know: HN and emojis…) as monastic marginalia for three hundred years and then maybe we’ll spare one code point for it.

Why shouldn’t emoji be an annually refreshed collection of fun fashionable clip art? That’s exactly how users use emoji - for culturally relevant references. As a user, I get excited to see all the new emoji each release!

The reason why this is a bad idea is that it's hard to remove emoji when they go out of fashion. You can't just pull characters out of the set without breaking the backwards compatibility assumptions made by every programming language that supports Unicode identifiers, and you can't change the meaning of characters without a painful, politically controversial collaboration between every large tech company (see the pistol emoji).

Even ZWJ sequences are forever: we're still stuck with the "eye in speech bubble" emoji despite the fact that it's a logo for a defunct anti-bullying campaign that doesn't even have a working website anymore.

Why does it matter, though? The symbols are already drawn, and the code that handles the emoji range is already written. As for "obsolete" emojis, regardless of the original intent, they will be appropriated in no time if some suitable use comes along; the official name in the Unicode chart is merely a historical curiosity (and, indeed, those names are wrong for many "legitimate" characters, but can't be changed for the same back-compat reasons).

Right, but Unicode also has ancient Egyptian hieroglyphics in it. Is it actually problematic to include a few more code points?

> it’s become an annually refreshed collection of fun fashionable clip art instead of an impartial repository of humankind’s symbols

Constraints are sometimes good. It's great to see how something like the Eggplant & Peach emoji got hijacked and used as a sexual reference.

All threads need at least one irrelevant pet peeve-aside.

So what? Unicode follows language and language is descriptivist not prescriptivist.

Nitpick: language is neither descriptivist nor prescriptivist, language simply is. A modern linguist is generally descriptivist in their approach to language.

Aside from that, I don't think the terms really apply to a discussion about emoji, because what makes emoji odd is that they are almost never in common usage as part of the language until Unicode adds them. That makes it very hard to claim that Unicode is describing anything in particular by adding emoji, and in fact makes their role much more prescriptive than not.

…yet they still don't think about adopting Tengwar.

Actually, only Klingon was officially rejected. The Tolkien languages never had a fully-featured proposal to Unicode, but space for them is even tentatively allocated in Unicode (https://www.unicode.org/roadmaps/smp/, U+160xx).

Klingon was explicitly rejected for a few reasons. Most prominently, there were serious concerns about Paramount asserting trademark rights over the characters. Especially at the time of its rejection, it wasn't clear that there was any organic use of the script, even among people speaking Klingon (note that in its earliest uses, it wasn't even used consistently as an alphabet for the language it is supposed to represent). Those reasons don't apply to Tengwar--and it's also been noted since then that the Unicode committee might look more favorably on a newer proposal for Klingon.

> Unicode really should have had a similar “no original designs” rule.

I think it does, doesn't it?

Here is some info on unicode rules for new characters, although there doesn't seem to be a single succinct statement, and there are slightly different standards for different categories ("letters" vs symbols vs emojis). But I am pretty sure you can't just make up a character and succesfully submit it to unicode on the argument that it would be useful, this thing you just made up; it has to have already established use to get into unicode, generally.

http://www.unicode.org/pending/proposals.html

Which characters are you annoyed by specifically?

It's true there are separate rules for emoji's, which are... weird. I seem to recall unicode initially tried to resist emoji's at least to some extent, before just giving in.

But emoji's are actually pretty clear, to me, as a success of unicode. People want them, they have achieved pretty widespread use globally, and if that's going to be so, we benefit from them being encoded in a standard vendor-neutral way. We're far better off with emoji's in unicode than, say, every vendor using their own non-standard use of private codepoints. (Unless you argue that if they weren't in unicode they wouldn't have caught on; I am sympathetic to the argument that we'd better off without emoji's, but I'm pretty sure if they weren't in unicode we'd still have them, just non-standardly).

Here are some of the guidelines for which emoji's will be encoded in unicode:

https://unicode.org/emoji/proposals.html#selection_factors

It's a bit fuzzy, but you generally can't simply get a new thing you just invented encoded in unicode on the argument that it's cool.

The private unicode range is often used for icons on many web sites, but downloadable fonts are considered optional on the web, so we're left with unintelligible hex codes instead, at which point we've regressed to a pre-accessibility era when images typically didn't come with alt text, but at least there was a fallback mechanism.

You should read an emoji submission form. You have to prove that the emoji would be culturally relevant and useful before it's added.

What’s the difference between fashionable clip art and humankind’s symbols?

It feels like there’s this bias where people think they know what’s relevant and what’s just nonsense from the younger generation or what have you.

Mind you, Unicode stresses me out. It’s trying to catalog something that’s impossible to because it’s arbitrary and limitless. And it’s doing it within the arena of software engineering where we really crave rules and clear boundaries.

But if we didn’t try, we’d be worse off. Unicode is in some ways a horror, but it’s valuable, even if flawed.

Andrew's Babelmap [1] is one of those applications that, if you do anything text or typography related, is basically required owning. With a donation, of course.

[1] https://www.babelstone.co.uk/Software/BabelMap.html

I'm usually ok with what macOS Character Viewer offers. I am rarely on Windows and didn't know about BabelMap. It looks like it fills the gap there.

I work mostly on Linux so I hacked a Character Viewer clone in Rust over a weekend recently[1].

It just does what I need but I'm planning to add features to it if I find them useful.

So I am curious: what functions does BabelMap offer that you can't live without, especially as a typographer?

[1] https://github.com/virtualritz/glyphana

Since you mentioned macOS, it would be remiss of me to not mention UnicodeChecker:

https://earthlingsoft.net/UnicodeChecker/index.html

I am a big fan of unicode.link[1] for inspecting existing strings, searching for codepoints, and comparing how strings get encoded.

[1] https://unicode.link/

Some random characters didn't render in my browser. Upon inspection:

font-family: Georgia, Serif;

I don't think those fonts support all of Unicode. Google created their Noto fonts [1] for this purpose; I wonder why those aren't being used.

[1] https://en.wikipedia.org/wiki/Noto_fonts

gentium is a font with a very large number of glyphs also

https://software.sil.org/gentium/

But only for Latin/Greek/Cyrillic scripts; it makes no claim to be a pan-Unicode font (family).

ok I did review the Gentium characters after this.. Gentium is an odd set of every dot, egrave, umlaut and circumflex, in many, many variations. You are right that Gentium does not make an effort to represent unicode code pages. Instead, it seems that the designers know about very old and odd language systems of Eastern and Central Europe, and have reproduced those.

Browsers will generally do "fallback" to some other font, if the font(s) named in the CSS don't support the characters present in the text. But for some of the rarer characters, you may not have any available font that supports them.

See https://en.wikipedia.org/wiki/Fallback_font. It typically isn’t a browser feature, but an OS one.

Since 1998 MacOS has a “last resort” font that has glyphs (not necessarily unique) for every Unicode code point. They donated it to Unicode (https://en.wikipedia.org/wiki/Fallback_font#Unicode_Last_Res...), so I expect most OSes running full-blown modern browsers to have it or something similar (those running smaller browser engines may be too space constrained to have room for it)

    http://www.chinaknowledge.de/Literature/Science/shuowenjiezi.html

I have the noto fonts and ctext dot org's hana fonts but still see tofu in the above page. Whatever font is used on the iPhone's Pleco app the correctness depends on context where you are in the app.

These two examples often are confused: 日曰

No single font supports all of Unicode, that would be highly cumbersome. The "Noto fonts" is really a family of a hundred or so fonts, each of which supports a separate part of Unicode (not necessarily a block).

I believe there are semi-exceptions to what I said, although you wouldn't want to use such a font for practical purposes. These exceptions are fonts that display the Unicode code point in a box, so they would display the ASCII 'A' as a box containing "0065" (I forget whether they use hex or decimal, or maybe there are both). I suspect these fonts create their glyphs on the fly.

I kind of expected this to be more an overview of the new stuff in Unicode 15.0. As the author of a Rust Unicode crate (finl_unicode), I always like to dig through the release notes to see what sort of strange new stuff is on offer.

Perhaps a better match for some folks' expectations, the Unicode Consortium's YouTube has plenty of talks on https://youtube.com/@unicode. Low view counts but often quite fascinating

GNU unifont has the entire MBP hut is a bitmap font. Is there an equivalent monospaced scalable font we can use in GPL software?

Noto Sans? https://fonts.google.com/noto/specimen/Noto+Sans+Mono

Unfortunately I also want technical symbols Depth, Counterbore, and Countersink and perhaps more.

Tangent:

I recognized the domain and tried to remember why, and now I remember.

I'm working on a game, and babelstone.co.uk has probably the world's most comprehensive (and high quality) set of runic fonts:

https://www.babelstone.co.uk/Fonts/

https://www.babelstone.co.uk/Fonts/Runic.html

https://www.babelstone.co.uk/Fonts/AngloSaxon.html

This is the most important part of Unicode for me:

https://www.unicode.org/reports/tr9/tr9-46.html

because I speak a right-to-left language. Whoever wants to write an application involving text entry, and truly support localization or internationalization, should take the time to read at least section 3:

https://www.unicode.org/reports/tr9/tr9-46.html#Basic_Displa...

I would love to see someone make an image to unicode "curve fitting" algorithm or converter, similar to ANSIDRAW.

See https://shapecatcher.com/.

And yet there is still no unambiguous lower case "I" or upper case "i".

thats the job of a font, not encoding

No, the fact that there is no codepoint that makes those mappings ambiguous is due to the way Unicode decided to save to codepoints for seemingly no good reason.

What should be _the_ value of `"I".lower()`? Or, "i".upper()?

And please don't bring up locales. The whole point of accepting the complexity of Unicode is to be able to take a document which stands on its own without external references.

> Early character encodings also conflicted with one another. That is, two encodings could use the same number for two different characters, or use different numbers for the same character.

> The Unicode Standard provides a unique number for every character, no matter what platform, device, application or language.[1]

Those statements are outright lies: Unicode does not provide a unique number fpr "upper case Turkish dotless i". Nor does it provide one for "lower case Turkish dotted i".

If it did, it would be possible to correctly map "i" to "I" or "İ" and "I" to "i" or "ı" without having to know anything other than the source codepoint.

The font does not even come into play here.

[1]: https://unicode.org/standard/WhatIsUnicode.html

Unicode is for representing text, not allowing arbitrary manipulation of it. It isn’t the job of Unicode to encode those relationships. Also, the Turkish `i` stuff is just the tip of the iceberg. Should Unicode be able to round-trip `'ß'.upper().lower()`? Keeping the existing capitalization of ß → SS, you need to define a "uppercase S that used to be ß" character. Then there’s the Dutch `ij`, in which both characters are either uppercase or lowercase (`Ij` at the start of a word is incorrect). There’s a ligature in Unicode, but it’s only for compatibility with some legacy keymaps. But is there a point in adding a new version of "S" that a lot of software would not recognize as equivalent to the plain old ASCII "S" (and one might end up far away from a ß due to copy-pasting or stuff), bringing weird bugs and security issues? Should the Dutch throw out all their keyboards just so they get a new key for the special IJ ligature?

> Keeping the existing capitalization of ß → SS, you need to define a "uppercase S that used to be ß" character.

Even more complicated, there is a capital form of ß:

https://en.wikipedia.org/wiki/%C3%9F

> Until 2017, there was no official capital form of ⟨ß⟩; a capital form was nevertheless frequently used in advertising and government bureaucratic documents.[10]: 211 In June of that year, the Council for German Orthography officially adopted a rule that ⟨ẞ⟩ would be an option for capitalizing ⟨ß⟩ besides the previous capitalization as ⟨SS⟩ (i.e., variants STRASSE and STRAẞE would be accepted as equally valid).[11] [12] Prior to this time, it was recommended to render ⟨ß⟩ as ⟨SS⟩ in allcaps except when there was ambiguity, in which case it should be rendered as ⟨SZ⟩. The common example for such a case was IN MASZEN (in Maßen "in moderate amounts") vs. IN MASSEN (in Massen "in massive amounts"), where the difference between the spelling in ⟨ß⟩ vs. ⟨ss⟩ could actually reverse the conveyed meaning.[citation needed]

No character encoding standard can save you from that kind of complexity. You need special-case code on a language-by-language basis.

The capital ẞ is a recent invention, it’s in Unicode since 2008. While it would solve the issue, most programming languages and other tools will convert it to SS (though things are changing; Gboard now offers an actual uppercase ẞ instead of the SS it had some years ago). I don’t know what actual humans think, but I suppose the no-capital-ß-exists position is ingrained in many German speakers.

I wasn't saying it was a solution, I was saying it was a further complication.

Also, Unicode didn't invent it. Germans invented it.

> The whole point of accepting the complexity of Unicode is to be able to take a document which stands on its own without external references.

What makes you think this is the whole point? I don't believe that "whole point" is actually possible with global human languagues as actually used, nor do I think those behind unicode historically or presently have considered this the "whole point".

Do you have any reference suggesting this was meant to be the "whole point" of unicode?

I think it's actually a pretty amazing success that unicode does give us algorithms for manipulating text in various ways, that actually work pretty darn well... but yes, they sometimes require locale parameters.

"Those statements are outright lies: Unicode does not provide a unique number fpr "upper case Turkish dotless i". Nor does it provide one for "lower case Turkish dotted i"." By design, and I would argue that this is the correct design.

What it does provide is a unique code point for "upper case I", "upper case dotted I", "lower case dotted i" and "lower case dotless i".

Yet, for example, it distinguishes upper and lower case versions iota[1] from "upper case i" and "lower case dotless i". A similar provision could have been made for upper and lower case versions of "Turkish dotless i" and "Turkish dotted i". But it wasn't.

That means "ι".upper().lower() produces "ι" but "ı".upper().lower() produces "i" (unless you do it on a computer set to a Turkish locale:

    >>> "ι".upper().lower()
    'ι'
    >>> "ı".upper().lower()
    'i'

Note we have:

    LATIN CAPITAL LETTER IOTA
    LATIN SMALL LETTER IOTA
    GREEK CAPITAL LETTER IOTA
    GREEK SMALL LETTER IOTA
    MODIFIER LETTER SMALL IOTA 
    TURNED GREEK SMALL LETTER IOTA
    CYRILLIC CAPITAL LETTER IOTA
    CYRILLIC SMALL LETTER IOTA 
    APL FUNCTIONAL SYMBOL IOTA
    MATHEMATICAL CAPITAL IOTA 
    MATHEMATICAL BOLD SMALL IOTA 
    MATHEMATICAL ITALIC CAPITAL IOTA 
    MATHEMATICAL ITALIC SMALL IOTA 
    MATHEMATICAL BOLD ITALIC CAPITAL IOTA 
    MATHEMATICAL BOLD ITALIC SMALL IOTA 
    MATHEMATICAL SANS-SERIF BOLD CAPITAL IOTA 
    MATHEMATICAL SANS-SERIF BOLD ITALIC CAPITAL IOTA 
    MATHEMATICAL SANS-SERIF BOLD ITALIC SMALL IOTA

That's lotta iotas[1]. There's more[2]:

    GREEK SMALL LETTER IOTA WITH DASIA
    GREEK SMALL LETTER IOTA WITH PSILI AND VARIA
    GREEK SMALL LETTER IOTA WITH DASIA AND VARIA
    ...

The point is that almost all of these variations on the theme with explicit duplicates get their upper and lower case distinct codepoints, yet "lower case Turkish dotted i" needs to for some reason map to 0x69 in ASCII and "upper case Turkish dotless i" needs to map to 0x49 ASCII.

It is stupid and directly contradicts the statement:

> The Unicode Standard provides a unique number for every character, no matter what platform, device, application or language.[3]

The statement on the Unicode.org web site cannot be attributed to ignorance.

[1]: https://en.wikipedia.org/wiki/Iota

[2]: https://www.unicode.org/charts/PDF/U1F00.pdf

[3]: https://unicode.org/standard/WhatIsUnicode.html

That can't be fixed. You just have to know if you're in a locale that wants one or another rule for that. Well, I suppose the UC could still introduce new codepoints for I and i that specifically for Turkish, but it wouldn't be convenient for users of Turkish.

Block 309 : Chess Symbols

Wow, Unicode seems to be expanding way out of the original goal of representing written characters - how often would these symbols be even used? And how many font sets would bother including them?

Wikipedia has some rationale: https://en.m.wikipedia.org/wiki/Chess_symbols_in_Unicode

Probably the most notable:

> Use figurine algebraic notation, which replaces the letter that stands for a piece by its symbol, e.g. ♘c6 instead of Nc6. This enables the moves to be read independent of language (the letter abbreviations of pieces in algebraic notation vary from language to language).

I have tool which lets you scroll through all characters in a large document. It also shows names, scripts, and IDNA-like status.

https://adraffy.github.io/ens-normalize.js/test/chars.html

I found this more entertaining than the new Avatar movie.

Reply on news.ycombinator.com