Hacker News Reader

Comment by pcwalton | Hacker News Reader

Sadly, the pervasiveness of JavaScript means that UTF-16 interoperability will be needed as least as long as the Web is alive. JavaScript strings are fundamentally UTF-16. This is why we've tentatively decided to go with UTF-16 in Servo (the experimental browser engine) -- converting to UTF-8 every time text needed to go through the layout engine would kill us in benchmarks.

For new APIs in which legacy interoperability isn't needed, I completely approve of this document.

View on HN

Yeah, it's really sad the number of legacy APIs which have standardized on UTF-16.

The Windows API calles UTF-16 "Unicode". Most Mac OS X APIs use UTF-16. JavaScript and Java both use UTF-16. ICU uses UTF-16. So while UTF-8 is technically superior in almost every way, it's going to be an uphill battle to standardize on it.

I appreciate that new languages like Rust and Go made the choice of UTF-8 as their native text encoding. But there's a lot of inertia for UTF-16, and I'm not sure it'll be easy to ever get free of it.

OS X APIs generally use NSString/CFString, which hide the actual encoding of the string; they can be any encoding at all internally.

While they could, in theory, hide the actual encoding, the APIs all refer to UTF-16 code units as "characters;" so while they could use UTF-8 as the internal encoding, you need to use and understand UTF-16 in order to interact with them properly. When you ask for the "length" of a string, you are told the number of UTF-16 code units. When you get a character at an index, you get the UTF-16 code unit. That's what I mean when I say the APIs use UTF-16; everything in the API that deals with individual "chracters" is actually referring to UTF-16 code units.

The same is true of JavaScript; while you could technically implement the strings however you want, the APIs are all oriented around UTF-16 code units. And the Windows API as well, is all built around UTF-16 code units.

The problem with all of these APIs is that they make the mistake of conflating characters and code units. They all make the assumption that a character consists of a single, fixed width integer, of some given size (16 bits in the case of UTF-16). It is better to distinguish between indexing in code units (such as bytes in UTF-8 or 16 bit integers in UTF-16) and indexing in code points, or glyphs, or whatever higher level concept you are talking about. Really, for anything higher than the code unit level, you should be dealing with variable-length strings, and not try to force that into fixed length units. With UTF-8, there's no temptation to treat a single code unit as being an independently meaningful entity, as that assumption breaks down as soon as you get past the ASCII range; while with UTF-16, it's easy to make that mistake, since it holds true for everything in the Basic Multilingual Plane, which contains most characters you're likely to encounter on a day to day basis.

Some old languages have also made that choice, albeit recently in Python's case:

http://www.python.org/dev/peps/pep-3120/

However, when it comes to the internal representation of text, things are quite complex as of Python 3.2: http://www.python.org/dev/peps/pep-0393/

ICU has some UTF-8 functionality. But ICU is really ugly.

http://userguide.icu-project.org/strings/utf-8

As long as you don’t care about errors, converting to UTF-8 is quite fast. Just use native calls:

  var decode = function (bytes) {
    return decodeURIComponent(escape(bytes));
  }
  var encode = function (string) {
    return unescape(encodeURIComponent(string));
  }

(Definitely test it before wailing about benchmarks. My guess is that whatever else you’re doing is likely much slower.)

If you do care about errors, or especially if you need to deal w/ UTF-8 streams that might be chopped mid-character, use something like https://github.com/gameclosure/js.io/blob/master/packages/st...

I think you misunderstand -- I'm referring to the actual systems-level implementation of the browser engine itself. I'm not talking about the implementation of web apps.

Consider a pattern like this: A page calls document.createElement(), adds a large text node (say, the collected works of Shakespeare in text form) to it, calls window.getComputedStyle() on that element, then throws the element away. This series of DOM manipulations must go through the layout engine. If the layout engine knows only UTF-8, then the layout engine has to convert the collected works of William Shakespeare from UTF-16 to UTF-8 for no reason (as it needs an up-to-date DOM to perform CSS selector matching for the getComputedStyle() call). There is no reason to do that when it could just use UTF-16 instead and save itself the trouble.

What part of that process requires UTF-16? JavaScript doesn't require UTF-16; it just requires Unicode. You could use UTF-8 in your JavaScript implementation as well.

> What part of that process requires UTF-16? JavaScript doesn't require UTF-16; it just requires Unicode. You could use UTF-8 in your JavaScript implementation as well.

Actually, JavaScript _does_ require UTF-16. From the ES5.1 spec:

> A conforming implementation of this Standard shall interpret characters in conformance with the Unicode Standard, Version 3.0 or later and ISO/IEC 10646-1 with either UCS-2 or UTF-16 as the adopted encoding form, implementation level 3. If the adopted ISO/IEC 10646-1 subset is not otherwise specified, it is presumed to be the BMP subset, collection 300. If the adopted encoding form is not otherwise specified, it presumed to be the UTF-16 encoding form.

JS does require UTF-16, because the surrogate pairs of non-BMP characters are separable in JS strings.

'<non-BMP character>'.length == 2 '<non-BMP character>'[0] == first codepoint in the surrogate pair '<non-BMP character>'[1] == second codepoint in the surrogate pair

Any JS implementation using UTF-8 would have to convert to UTF-16 for proper answers to .length and array indexing on strings.

Still doesn't mean that it has to store the strings in UTF-16. And a bit of common-case optimizing would allow ignoring that case until a string actually ends up with a non-BMP character in it.

Alternatively, if a JavaScript implementation chose to completely ignore that particular requirement, I'd guess that approximately zero pages (outside of test cases) would break, and a few currently broken pages (that assumed sane Unicode handling) would start working.

This is horrifying. :-(

If stuff like length and indexing depend on the actual underlying representation, the implementation is broken regardless of the actual format used.

Change "implementation" to spec and I agree with you.

An implementation that fails to meet the spec can be argued as not broken, even if said spec is broken.

I agree with you in concept, but the Web is actually not strictly tied to javascript. In the short term (say, next decade), JS is probably not going anywhere, but in the longer term I hope someone creates a more well-thought-out replacement. (And for the record, I kind of like javascript, just a few things I would change with it.)

Yes, like a proper VM that we can compile our preferred language to. It is ridiculous that the "language of the web" is something that is forced upon us instead of chosen. The number of languages that compile to JS illustrate the point nicely.

The likelihood of Javascript becoming non-dominant in the next 20 years is pretty low, I think. Rather than replacing it wholesale, you'll simply see it turn into something better over time.

A recent discussion about this involving Brendan Eich and a few other JS folk: https://gist.github.com/1850768

> converting to UTF-8 every time text needed to go through the layout engine would kill us in benchmarks

So what? Is your goal to create useful software, or win at worthless benchmarks?

Well, we're talking about DOM manipulation performance here. Pages that use DOM manipulation heavily will see a potentially-unacceptable performance loss if text always has to be converted to UTF-8.

Is fast DOM manipulation important? Given that the only way for the sole scripting language on the Web to display anything or interact with the user is through DOM manipulation, I think it's worth optimizing every cycle...

http://www.utf8everywhere.org/#faq.cvt.perf

If the function you're calling with UTF8 is non-trivial, converting a few dozen bytes is unlikely to make a significant difference. Benchmark it, of course, but don't be surprised if you don't need to care. Modifying the DOM is probably going to be non-trivial.

UTF-8 -> UTF-16 conversion was at one point a noticeable fraction of Firefox's startup time.

https://bugzilla.mozilla.org/show_bug.cgi?id=506431

Since then we've done things such as fast-path ASCII -> UTF-16 conversion with SSE2 instructions. Converting a few dozen bytes is unlikely to make a significant difference, but often one needs to deal with more than a few dozen bytes.

I believe the authors wrote it clear enough that they don't rule out UTF-16 completely:

> We believe that all other encodings of Unicode (or text, in general) belong to rare edge-cases of optimization and should be avoided by mainstream users.

So if you're writing a browser that must use UTF-16 in its Javascript engine due to dumb standards... it is a reasonable performance optimizations to use UTF-16 for your strings. But how many people write Javascript engines?

It's not the conversion that kills you, it's the memory allocation. If you're trying to interact with a "chatty" UTF-16 API, the difference between using UTF-8 and UTF-16 in your implementation could the difference between passing a pointer (a cycle or two) and allocating/deallocating a buffer (potentially hundreds of cycles) per call.

There are obviously ways around this. The API could be rewritten to exchange strings less frequently or use static strings that could be replaced with handles. You can try to be clever about your buffer allocation and share one amongst all calls (but watch out for threading issues!) You could write your own allocator. But all this plumbing just increases complexity and the risk of bugs, along with adding its own performance cost.

I'm not arguing against UTF-8 as the preferred encoding for many future applications, but the "minimal overhead" example given in the manifesto isn't particularly convincing.

I agree in principle. But "chatty" APIs usually work with short strings, in which case you can use stack allocations in those performance critical calls.

See my comment above; I can construct cases that make this sort of conversion have unacceptable overhead.

Would it matter in a real-world setting? I can't say for sure, because nobody I know of has tried making a production-quality UTF-8 web layout engine. But, in my mind, none of the benefits of UTF-8 (memory usage being the main one in a browser [1]) outweigh the performance risks of doing conversion. And the risk is real.

[1]: Note that you still need UTF-16 anyway, for interoperability with JavaScript. So using UTF-8 might even lead to worse memory usage, due to the necessity of duplicating strings, than a careful UTF-16-everywhere scheme that takes advantage of string buffer sharing between the JS heap and the layout engine heap would.

Is string storage really that big a portion of the browser's memory usage? I find it hard to believe that my browser is currently storing nearly a gigabyte of text right now I'm not saying that there wouldn't be performance overhead, but would it be significant? I would be surprised if it made a big difference.

For legacy reasons, there probably isn't a point in changing it. But I'd be surprised if performance reasons turned out to be gating.

Claims like this are exactly the reason why we will still be stuck with multiple useless encodings in 2030.

I wonder how long it will take until people find their balls and decide to move towards the right direction.

Reply on news.ycombinator.com