Unicode: The people who made the internet open to anyone

I learned almost nothing about Unicode as part of my CS degree. When I graduated, all I was aware of was that it existed and I thought that it used just 16-bit characters, which was the original Windows Unicode implementation.

That wasn't exactly accurate, and I didn't have any idea of Unicode code points and the different ways of encoding them (UTF-8, UTF-16, UTF-32, and whatever else).

It wasn't until later when I decided to write a C++ library to deal with Unicode strings did I learn about code points, code units, UTF-8 and UTF-16.

https://github.com/Maultasche/UtfString

The library essentially re-implements the std::string interface, but deals with code points instead of bytes. It'll successfully handle code points of variable size (1 to 4 bytes for UTF-8 and 1 to 2 16-bit values for UTF-16).

Sorry, the documentation needs to be converted to Markdown. It's currently in HTML that gets fed to doxygen. I wrote this before markdown and Github existed.

Here's an interesting thing I learned while implementing the library. UTF-16 is no longer consists of just 16-bit characters. It did originally, but the number of Unicode code points became too large for 16 bits. Like UTF-8, a single code point (which often, but not always, has a 1-1 relationship with a character), can have a variable number of code units. UTF-16 code points usually consist of a single 16-bit value (aka code unit), but the code points higher up on the Unicode map will consist of two 16-bit values.

If I hadn't done the work and research to write the library, I probably would still only have a vague idea of Unicode and its encodings.

I think they mean it that Unicode as a technology should be invisible to the average user. Just as no one should be forced to think about whether numbers above 2 billion cause trouble, no one should be forced to remember the differences between Latin 1, UTF-8, UTF-16LE, etc. pp. Normal people just want to write their language and people expect to see what they wrote. If they see question marks, or diacritics next to their respective letters, or completely different scripts, then things failed. But I'd argue that normal people do not need to know how or why that failed. Even for developers with a bit of Unicode experience it's often not trivial to recover text broken in such ways.

Oh, and to answer your question: I never learned anything about Unicode in uni either. As for myself, I've been lurking on the Unicode mailing list since about 2008.

Regular expressions strike me as a far more fundamental CS concept than Unicode.

The first and third dissents are dumb. Basically the attack is that, because the glyphs used to represent the character look the same but are actually different code-points they have potential to be abused.

But there's no reason why this would only be limited to Unicode. Any other encoding system that keeps a cyrillic alphabet in a separate code range from the latin one would have this problem. And it's much less efficient for your text chomping program that reads cyrillic to check all of a character range except for the 2-3 that also have existing latin characters that look the same as the cyrillic ones.

And I really don't understand why "I Entity Unicode" is even a valid complaint. Or what it's even complaining about If your font doesn't have a glyph that matches your character, it's not going to show up no matter which encoding scheme you use.

Unicode got alot of buy-in from publishers as it met all of their needs. The tech companies followed suit, eventually, which is a heck of alot better than the dark ages when each computer company had their own custom platform for these encodings in the U.S. alone. And then there's individual solutions to the problem overseas where the wheel was reinvented for each language there.

Indeed. Now that the Astral Plane has opened up, I think it is past time for Unicode to reconsider its decision on conlang alphabets and include things like Klingon, Tengwar, Aurebesh, and similar. I suppose coming up with criteria for conlang alphabet significance might be complicated, but certainly similar scholarship qualifications might be used as some of the dead languages.