Comment by danso | Hacker News Reader

Comment by danso | Hacker News Reader

Parent Full thread

> These articles answered a lot of my questions, but they also raised a new one. Every one of them mentions something called Unicode. Some of them in passing, as if to say, “We all know about Unicode, right? Let’s move on.”

Anyone who came through with a typical CS/CompEng degree: did you learn about Unicode in any depth? I didn't. We didn't even cover regular expressions. I grew to hate Unicode but only recently have learned to respect what a monumental achievement they were. I mean, I would hate buffer overflows (more) if I didn't understand pointers and memory...but after understanding why they (pointers and memory) are the way they are...They are painful, but I don't suffer :)

This remark by Chris Granger always bothered me:

> We don't want a generation of people forced to care about Unicode and UI toolkits. We want a generation of writers, biologists, and accountants that can leverage computers [1]

[1] http://www.chris-granger.com/2015/01/26/coding-is-not-the-ne...

I don't think people should be forced to decode Unicode by hand, or even memorize little/big endian. But to not understand how fundamentally text translates into numbers? That, from now on, humans will have a normalized way of communicating with each other, whether by Internet or starship viewscreen, for as long as the standard exists?

Unicode is annoying, but only in the sense that going to France and not speaking French is totally annoying.

I learned almost nothing about Unicode as part of my CS degree. When I graduated, all I was aware of was that it existed and I thought that it used just 16-bit characters, which was the original Windows Unicode implementation.

That wasn't exactly accurate, and I didn't have any idea of Unicode code points and the different ways of encoding them (UTF-8, UTF-16, UTF-32, and whatever else).

It wasn't until later when I decided to write a C++ library to deal with Unicode strings did I learn about code points, code units, UTF-8 and UTF-16.

https://github.com/Maultasche/UtfString

The library essentially re-implements the std::string interface, but deals with code points instead of bytes. It'll successfully handle code points of variable size (1 to 4 bytes for UTF-8 and 1 to 2 16-bit values for UTF-16).

Sorry, the documentation needs to be converted to Markdown. It's currently in HTML that gets fed to doxygen. I wrote this before markdown and Github existed.

Here's an interesting thing I learned while implementing the library. UTF-16 is no longer consists of just 16-bit characters. It did originally, but the number of Unicode code points became too large for 16 bits. Like UTF-8, a single code point (which often, but not always, has a 1-1 relationship with a character), can have a variable number of code units. UTF-16 code points usually consist of a single 16-bit value (aka code unit), but the code points higher up on the Unicode map will consist of two 16-bit values.

If I hadn't done the work and research to write the library, I probably would still only have a vague idea of Unicode and its encodings.

I think they mean it that Unicode as a technology should be invisible to the average user. Just as no one should be forced to think about whether numbers above 2 billion cause trouble, no one should be forced to remember the differences between Latin 1, UTF-8, UTF-16LE, etc. pp. Normal people just want to write their language and people expect to see what they wrote. If they see question marks, or diacritics next to their respective letters, or completely different scripts, then things failed. But I'd argue that normal people do not need to know how or why that failed. Even for developers with a bit of Unicode experience it's often not trivial to recover text broken in such ways.

Oh, and to answer your question: I never learned anything about Unicode in uni either. As for myself, I've been lurking on the Unicode mailing list since about 2008.

Sure, Unicode itself should be invisible, because Unicode is the solution to the problem. The problem itself is having tons of incompatible encodings in use, and people do have to understand what an encoding is if they're from a place where "legacy" encodings (i.e. ones that aren't Unicode encodings) are still in use. My mind usually leaps to Japan's continued adherence to SJIS, but I'm sure there are others as well.

Regular expressions strike me as a far more fundamental CS concept than Unicode.

the CS concept of regular expressions is not the same as the as implemented perl-compatable-regular-expression engine or posix regex which is what 99.9% of references to regular expressions are.

They are not any more a fundamental CS concept than the syntax of the C language is, or unicode encoding.

PCRE/posix regex arn't even regular expressions in CS terms (due to stuff like backreferences, and in the case of PCRE even turing completeness).

Reply on news.ycombinator.com