Hacker News Reader

Comment by int_19h | Hacker News Reader

To elaborate: UTF-8 is the most common I/O encoding, but UTF-16 is often what you need to work with the OS APIs (Win32, macOS) or popular frameworks (Qt).

View on HN

The funny thing is that C++ sort of has a flexible string class which uses system-dependent conversions, std::filesystem::path

It's super convenient for working with Windows file APIs, but everything else is still a pain.

Yep. There are still some annoyances like invalid UTF-16, so maybe there is more is needed than just the basic transformations — some precise handling or parameters specifying how to handle these edge cases could be necessary.

Handling UTF-8 versus WTF-8, and properly round-tripping these representations, is probably a bigger issue than graphemes and canonical forms for most users.

Also unfortunately the web browser. Javascript strings are UTF-16 (Well, UCS2).

This used to be the case when JS was created. However, I don't think that this is true anymore. It's definitely not under the hood. Nor, if I recall correctly, if you attempt to convert them from/to typed arrays. Maybe if you use `fromCharCode`?

JS strings are exposed as UTF-16 in the language — surrogate pairs have length 2 and you can index into half of them.

A JavaScript string is conceptually an array of 16-bit unsigned integers (https://tc39.es/ecma262/multipage/ecmascript-data-types-and-...). String indexing and substring operations treat strings this way, and they couldn't change the model easily without breaking code. If you index a string containing an emoji with a code point greater than U+FFFF, you get string containing an unpaired surrogate code unit, which makes the string UCS-2, not UTF-16. I don't know how it's implemented memory-wise in the various JavaScript engines and am not familiar with typed arrays.

> It's definitely not under the hood.

Really? I'd love to see some references to that.

I know there's some operations to convert a javascript string to UTF8, but everything which measures a string counts using 2-byte UCS2 items. For example, array-style indexing, string.length, str.charCodeAt(), split(), slice(), indexOf(), etc.

Also lots of DOM APIs use those weird string lengths too. For example, input element selectionStart / selectionEnd fields measure the selection range by counting surrogate pairs, not characters.

> Really? I'd love to see some references to that.

I remember that SpiderMonkey shifted to multiple encodings around 2019 and v8 had done that earlier.

Source: I was part of the SpiderMonkey team when the shift happened.

Reply on news.ycombinator.com