Hacker News Reader

Should new programming languages built around internal use of UTF-8 be opting their binaries into this behavior by default?

(I'm mostly thinking about Rust, which I think has to do some contortions on Windows. Probably also Go, to the extent it uses native APIs; arguably Python too.)

View on HN

The Rust standard library has contortions in order to support invalid UTF-16. To do this it uses an encoding called "WTF-8"[0], which is an extension to UTF-8.

If the standard library dropped this requirement and only supported valid Unicode then it could simply use normal UTF-8 instead of WTF-8.

[0]: https://simonsapin.github.io/wtf-8/

The problem right now is that forcing the codepage to UTF-8 only works on Windows 10 build 1903+. So quietly forcing this opt-in may result in situations where everything works fine for the devs, but some of their users see breakage for no clear reason.

Long-term, though? Absolutely.

Reply on news.ycombinator.com