[citation needed]
UTF-8 is used for storage and serialization, yes, but most mainstream programming languages store unicode strings in memory as UTF-16. The notable exception is Rust, which does indeed use UTF-8 even in memory.
This is increasingly frequently false. For stupid historical reasons, far too many languages adopted potentially-ill-formed UTF-16 semantics (never well-formed, sadly), but this makes for such a bad representation that they’ve mostly abandoned it as a sole representation, in favour of more complicated arrangements. For example, Java strings can internally be single-byte Latin-1 or two-byte UTF-16. JavaScript engines do similar tricks these days, and although Python 3 just never did UTF-16 at all (it uses code point semantics, which is almost worse than UTF-16 semantics) it does a 1-byte/2-byte/4-byte hybrid representation thing too.
For one of the most extreme examples of becoming unglued from UTF-16, the Servo browser engine manages to use WTF-8 (UTF-8 plus lone surrogates, to allow representing ill-formed UTF-16) despite being forced by these historical reasons to expose UTF-16 semantics, and although the performance of some operations are harmed by it, others are improved, and in the balance it was a pretty convincing win when they tested it (which many people did not expect).
PyPy is also a great example of changing your encoding to one that has mismatched semantics: it applies UTF-8 (or I suppose it must actually be UTF-8 plus surrogates) to Python, and likewise they found it surprisingly good for performance.
These things give me hope that even environments like the JVM and .NET CLR might eventually manage to switch their internal string representations to UTF-8 or almost-UTF-8.
But also I emphasise that these are historical languages that are stuck with the horrendous 16-bit decisions of the early-to-mid-’90s. When you look at newer languages, they almost always choose something more sensible, and UTF-8 is almost always at the heart of it.
What's so bad about storing a unicode string as a series of codepoints? UTF-8 is also a series of codepoints in essence.
> When you look at newer languages, they almost always choose something more sensible, and UTF-8 is almost always at the heart of it.
I still think that UTF-8 for memory is a bad idea. For example, it wastes a lot of bits to make sure that a string can be decoded starting from an arbitrary byte offset. This is generally a desirable property for files and some networking applications, but it's an absolute waste of space when used in memory.
>>> '\udead'.encode('utf-8')
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
UnicodeEncodeError: 'utf-8' codec can't encode character '\udead' in position 0: surrogates not allowed
>>> '\udead'.encode('utf-16')
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
UnicodeEncodeError: 'utf-16' codec can't encode character '\udead' in position 0: surrogates not allowed
—⁂—
> For example, it wastes a lot of bits to make sure that a string can be decoded starting from an arbitrary byte offset.
If you ditch self-synchronisability and ASCII purity of extension by ditching the first two bits (“1-0”) on continuation bytes, you can encode some more briefly: roughly, U+0800–U+1FFF go from three-byte to two-byte, and U+10000–U+FFFFF go from four-byte to three-byte, leaving just U+100000–10FFFF as four-byte. But honestly the difference isn’t often that much (unlike UTF-16 → UTF-8, which roughly halves memory usage most of the time, commonly even saving when dealing with text that makes heavy use of other scripts that are more compact in UTF-16 than in UTF-8 since you commonly mix things in with ASCII markup), and if you’re really trying to shave memory, other techniques like compression are far more effective.
I love microoptimisation (and spent a large number of hours on my Casio GFX-9850GB PLUS, where having only 30KB of memory encouraged learning to shave single bytes here and there!), but there’s also a lot to be said for consistency. Yes, some purposes could have an encoding that is more efficient than UTF-8. But doing this complicates matters quite a bit, and I think we’re better off with just UTF-8.
Also I will note that the self-synchronising nature of UTF-8 is useful for some operations: you can scan through strings much more quickly that way. So it’s not pure waste as a memory representation.
Go also uses UTF-8 btw.
I think you're being needlessly pedantic (I know it's unusual for HN).
Uh. What languages do you have in mind? C/C++ don't have any preference for UTF-16. Python3 doesn't use UTF-16. As you mentioned, Rust doesn't.
C/C++ don't have a preference but, for example, if you're writing for Windows, you'd probably want to use wide characters. I'm not sure what kind of strings Linux GUIs use but I suspect it's wide characters as well.