If you use Go, Rust, Python 3, or Javascript, this is a non-problem, because those are all native UTF-8. Only for legacy languages is this still a problem.
Go strings aren’t actually UTF-8; they aren’t even Unicode: they can contain arbitrary bytes. In practice, most strings will be UTF-8, but you can’t depend on it at all.
Rust strings are strictly UTF-8. I appreciate this greatly.
Python 3 strings are sequences of Unicode code points (as distinct from Unicode scalar values), allowing ill-formed Unicode (unmatched surrogates), and its internal representation is a disaster because they decided indexing by code point was worthwhile (it’s not, given the costs), so since CPython 3.3 strings are encoded as ISO-8859-1 (only able to represent U+0000 to U+00FF), UCS-2 (further able to represent U+0100 to U+FFFF) or UCS-4 (able to represent all Unicode code points).
JavaScript strings allow ill-formed Unicode, and the public representation can be described as either UCS-2 or ill-formed UTF-16 (almost all string access techniques work in UTF-16 code units, though new stuff is now mostly working in Unicode and UTF-8 terms). I think all major engines now use an internal representation of ISO-8859-1 if the string contains only U+0000 to U+00FF, or ill-formed UTF-16 for anything else; but Servo’s work has demonstrated that it’s possible to shift to WTF-8 (UTF-8 plus allowing unmatched surrogates), saving lots of memory on some pages and simplifying and slightly speeding up some things, but at the cost of random access performance by code unit index.
Go strings are supposed to be UTF-8, but it's not enforced.
Javascript - bleah.
No, it never looked like it was UTF-8. It did look like it was Unicode (of an unspecified encoding), but Python allows unpaired surrogates, which makes for ill-formed Unicode. Python lets you have '\udead'. Well-formed encodings of Unicode (such as UTF-8, UTF-16 and UTF-32) cannot represent U+DEAD.
There’s a big difference between “Unicode” and “UTF-8”.
> PyPy, though, finally went UTF-8.
Oh cool! I remember vaguely hearing the idea being mulled over, years ago. Good to see it happened: https://morepypy.blogspot.com/2019/03/pypy-v71-released-now-....
Wonder what they’ve done about surrogates; the only way they can truly have switched to UTF-8 internally is if they’ve broken compatibility, which I doubt they’ve done; I’m guessing that they actually use WTF-8, not UTF-8.
(In the meantime, I’ve changed “Python 3.3” in the grandparent comment to “CPython 3.3” for accuracy.)
https://stackoverflow.com/questions/12053168/how-to-properly-output-a-string-in-a-windows-console-with-go
https://github.com/intellij-rust/intellij-rust/issues/766
https://bugs.python.org/issue44275