In other words, to what extent are surrogate pairs a UTF-16 thing, rather than a Unicode thing that exists to accommodate for UCS-2 -> UTF-16?
In other words, to what extent are surrogate pairs a UTF-16 thing, rather than a Unicode thing that exists to accommodate for UCS-2 -> UTF-16?
https://simonsapin.github.io/wtf-8/
To be clear, this is not an official Unicode spec. It's a hack (albeit a pretty natural and obvious one) to deal with systems that don't do Unicode quite right.
I recently came across some old code that narrows wchar_t to UCS-2 by zeroing out the high-order bytes. Even though my test was careful not to generate any surrogates in the input, they showed up in the output when a randomly generated code point like U+1DF7C was mangled into U+DF7C.
A corrupted value like that is not necessarily a great example of something you want to preserve, but it's the sort of thing that late 90s code assumed about Unicode.
> The definition of UTF-8 prohibits encoding character numbers between U+D800 and U+DFFF, which are reserved for use with the UTF-16 encoding form (as surrogate pairs) and do not directly represent characters.
They could be replaced by the replacement character to produce a valid string.