Hacker News Reader

Comment by delamon | Hacker News Reader

We still have about 85% of codepoint space unused. Hopefully, by the time it becomes a problem, UTF-16 will be long dead

View on HN

i hope so too, but UTF-16 being used by languages such as java and javascript makes me fear it might be here to stay.... i hope im wrong

The number of glyphs available by adding additional bytes drops exponentially because each subsequent byte has one less bit available.

So I think if we ever were in a situation where > 1 million code points isn’t enough, then we should look at an entirely new way to serialise those code points.

I don't quite get it. 5-byte utf-8 encoding gets extra 5 bits compared to 4 byte, and 6-byte gets extra 10 bits. If you were thinking about bits in leading byte, then yes, you are losing one bit for every extra trailing byte, but you also get 6 bits from it. So adding a byte gives you extra 5 bits.

Yeah, you’re right. I might have attempted to do mental arithmetic before coffee…

Utf-16 is famously used by Windows for everything important as well.

And UTF-8 isn't even fully compatible with windows UTF-16 - UTF8 can't encode a lot of truncated windows UTF-16 filenames.. You need WTF-8 for that.

https://artoria2e5.github.io/XB18030/

It seems when designing Unicode most energy went into emoji. And there was nothing left for fancy things like fixed-length string buffers. The only explaination why UTF8 Buffers aren't compatible with UTF16 Buffers... is a really strong emoji...

Thankfully, this is changing. Win32's 'A' ANSI/ASCII APIs now support UTF-8 if your app declares such a wish. https://stackoverflow.com/a/69181417/1350209

The internal string encoding of a programming language doesn't matter as long as it supports UTF-8 at the boundaries. E.g. the text encoding standard on the web is clearly UTF-8, even though JS strings may be internally stored as UTF-16 (or any other encoding).

Same on macOS/iOS btw: AFAIK NSString is internally UTF-16, but I've never seen a UTF-16 text file on macOS, it's all UTF-8 (unless the file originated on Windows of course).

The internal string encoding and its limitations does leak into the APIs.

The problem is the boundary with, say, windows. In rust they have WTF-8 more or less just to deal with windows filenames.

But by then, the 4-byte limit of UTF-8 will itself have ossified. Even today, reverting back to the 6-byte limit is nigh impossible.

By then we will have quaternary quantum computers and FTL circuits where the information appears request it before you

Reply on news.ycombinator.com