Hacker News Reader

Comment by GolDDranks | Hacker News Reader

Excellent post, thank you!

By the way, I've sometimes heard that it was the popularity of emoji that finally made American and Western European coders learn about the particulars of unicode. Is this just a jokingly told anecdote, or did emoji really help?

View on HN

Emoji may have helped popularize support for Unicode characters outside of the Basic Multilingual Plane (The first 2¹⁶ code points in Unicode), because all but a few fall outside of it. Before emoji any code point outside of the BMP tended to have a niche audience. When I wrote my master's thesis in 2009, support for non-BMP characters was still sketchy (although XeLaTeX and PDF-viewer Evince could handle them neatly on the Linux desktop).

I doubt that emoji helped further popularize Unicode in Europe, because in my experience (anecdotal of course) developers were already well aware of the benefits of Unicode (and UTF-8 in particular for the web) before emoji got adopted from Japanese telephones and embraced en-masse. I suspect that in Europe the need for Unicode arose sooner. For Dutch for example; with the introduction of ISO-8859-15 in 1999 (subtly different from ISO-8859-1) you now often had two character encoding standards in mixed use for a single language, and when you had to render documents with names of people from various countries, things just got confusing (in the US diacritics tend to get dropped instead). UTF-8 offered a clean break from this.

It seems plausible that emoji may have helped Americans break the remnants of the ASCII barrier though.

(where "niche audience" includes speakers of the most spoken language in the world, but I assume you're talking about the West specifically)

I was under the assumption that for the average Chinese speaker the Basic Multilingual Plane sufficed. Chinese characters (as used in Chinese, Japanese, Korean, and Vietnamese) were certainly added beyond the BMP as well (the CJK Unified Ideographps Extensions), but to my knowledge these all tend to be rather rare (think characters from Buddhist sutras, linguistic research, historical use, et cetera, but also from names not yet provided for in the BMP).

Huh. IIRC they're a bit more common than that, but yes, not very common.

Or perhaps you're just talking about Chinese where I'm also considering Kanji and others. Which isn't from the most spoken language in the world, but still, top 10!

Actually for Japanese in particular the extended CJK ranges are of limited use (I'm more familiar with Japanese than Chinese), although they do contain the parts of JIS X 0213 that were still missing from Unicode (all of them rare characters) and some ideographical component parts that are useful for dictionary and educational software. With the latter you can for example use Unicode codepoints for the distinct components of a single Chinese character (e.g., 𠂉 as the top-right part of 族). That is a fairly specialised use though.

For Korean and Vietnamese the extra Chinese characters might apply to ancient names (geographical usually I would wager), but neither language has a lot of ideographs in modern usage.

It's certainly an interesting topic though.

Western European? I don't know - there are many languages in that part of the world which use some non-standard characters. Germany (not sure if it's still the case), France, Austria, Italy, Spain... Anyway, I always thought UNICODE ignorance is strictly an anglo-sphere thing - most of the rest of the world is very aware.

Western Europe uses more characters than ASCII like umlauts (äöü), accents (àèé), €¢, German ß and a few more, but large parts are still comfortably stuck in a codepage-style mindset thinking "ISO Latin-1 is good enough". (Also know as or confused with ISO/IEC 8859-1, CP-1252 or "Ansi".)

https://en.wikipedia.org/wiki/Code_page#ISO.2FIEC_8859-relat...

Central European languages can get more or less by with Latin-1, although even there many characters are already missing, iirc French, Czech, Dutch and Finnish. Special characters of many regional dialects are missing as well.

FWIW (more or less proper) Unicode support seems to be pretty much an implicit, hard requirement today for almost anything facing users.

Many government databases still just use some codepage and won't be changing anytime soon though. You can't get your name in a Swiss passport without first changing it into something ISO 8859-15 can represent.

http://www.24heures.ch/suisse/Les-noms-de-l-Est-mutiles-lors...

I find it pretty accurate, but that's just anecdotal from how I've seen software I use change in that time period, and how I've seen attitudes of people online change.

I'm sure plenty of Western programmers were aware of this stuff, though. Just not enough.

Reply on news.ycombinator.com