Hacker News Reader

Unicode looks like a big over engineered standard that had 50 hands trying to put their mark in

It looks like that because Unicode is trying to solve a problem that everyone thinks is easy until they uncover the true extent of encoding human languages.

How does this explain surrogate pairs?

Surrogate pairs were new to Unicode 2.0. Unicode 1.0 didn't anticipate the need for more than 65,536 code points (who would ever need more?); the main perceived threat to that limit having been resolved by Han unification.

Ok, but that doesn't answer the question; it's more of an indication that those design(at)s didn't uncover "the true extent" until years later

That's because Unicode chose to be a superset of all other encodings, so they've brought everyone else's complexity and tech debt.

Technically, a superset would have to somehow Schrödinger's cat around \ in latin1 and ¥ in Shift-JIS being the same codepoint.

Unicode just took it upon themselves to reliably round-trip legacy text... thus the precomposed forms.

Most of the other complexity and technical debt is in the writing systems themselves.

Reply on news.ycombinator.com