How UTF-8 Works

And yeah this problem only came later after UTF-8 was already invented. It's a smart solution also because at the time a lot of detractors of Unicode were opposed to the extra data requirements of the long form representations.

ASCII is itself valid utf-8, because ASCII is a subset of utf-8. But a multi byte encoded codepoint in UTF-8 cannot be confused with ASCII, because the highest bit is set in all the octets.

The output of files on disk can be UTF-8. The continued use of UCS-2 (later revised to UTF16) is happening in the runtime because things like the Win32 API which C# uses is UCS-2. The internal raw memory of layout of strings in Win32 is UCS-2.

*EDIT to add correction

The biggest problem is when you're working in an ecosystem that uses a different encoding and you're forced to convert back and forth constantly.

I like the way Python 3 does it - every string is Unicode, and you don't know or care what encoding it is using internally in memory. It's only when you read or write to a file that you need to care about encoding, and the default has slowly been converging on UTF-8.

That is so far from the truth.

I was delighted recently to stumble on a history of modern women course HIST1158 named Liberté Egalité Beyoncé and I immediately thought two things: 1. Why are our Computer Science courses given unimaginative names? and 2. What a useful test input, I bet some of our systems don't work correctly for this input even though an acute accent is hardly a bleeding edge feature.

I haven't been able to interest any Computer Science professors in fun names for their courses, but I was able in my test environment to name a COMP series course "Untitled Course Name" with a description explaining that "It is a lovely day in the village and there are only two hard problems in Computer Science".

What about UTF-256? Maybe not today, maybe not tomorrow, but someday...

E.g., Ruby runtimes scan the bytes in a string and then cache data about them in value called a code range. Knowing the code range, you can optimize many operations to not require additional linear scans of the string. Knowing a UTF-8 string consists only of ASCII characters can allow operations to be just as fast as if the string truly were ASCII-only (Ruby supports 100+ string encodings). And that fact is used throughout the core library to provide fast implementations of many operations (upcase, downcase, capitalize, gsub, substring, and so on). Moreover, a JIT can generate extremely tight code in those situations. Having to take a linear pass through the string to discover codepoint boundaries incurs a huge performance cost. While all strings could be treated uniformly and use Unicode tables for case mapping and such, the extra overhead is brutal. It has a measurable impact on string-heavy applications, such as template rendering and text processing.

In the most general case, yes, you know nothing about the string and can't make any assumptions. You can't even be sure the byte sequence is valid UTF-8. But, very often you do know properties of those strings. And you can manage boundaries where strings with known properties are joined with strings with unknown properties (e.g., variable interpolation in a template file).

Source code manipulation is frequently Unicode aware but doesn't care about combinations or things outside of a strict subset of Unicode to modify lexing control flow.

Being able to store (and later refer to) character offsets in the source code is a plus because they'll only ever occur in places where the strict subset is enforced.

This is especially true of languages with line-only comments, etc, where different writing systems being used won't affect the error message information.

Like I said, there are a few useful cases where having a fixed width encoding is beneficial. It's less helpful to the discussion to assert you know better for every case, ever.

My team managed a system that did a read from user data, doing input validation. One day we got a smart quote character that happened to be > U+10000. But because the data validation happened in chunks, we only got half of it. Which was an invalid character, so input validation failed.

In UTF-8, partial characters happen so often, they're likely to get tested. In UTF-16, they are more rarely seen, so things work until someone pastes in emoji and then it falls apart.

           10000011 11101001 U+E9 "é"
  10000001 10000011 11101001 U+10E9 "ჩ"

With a code like this, your dumb terminal can be built so it just automatically works. Without it, the terminal may be off by some number of bytes, and you have to have some sort of other (probably manual) synchronization procedure.

Similarly, imagine you've dialed in over modem to a remote system. You get some errors or lost text due to noise on the phone line. Or some bytes are lost due to bad RS232 flow control or no flow control. Your terminal emulator would be able to recover.

There might also be uses where text is broadcast, such as in TV closed captions, although maybe those have their own framing so you know where strings begin.

Maybe not in your bubble, but in my bubble this is the common case and highly optimized components using SIMD parsing is the exception.

By byte-by-byte, I presume you mean logically, not invoking a syscall for every byte. A parsing library is often handed a buffer to parse. If every layer had it's own buffering, you'd end up with too much data copying, which compounds and can spill your CPU caches in ways that microbenchmarks won't reflect, particularly in a streaming pipeline. So you stick to simpler, straight-forward code and data paths, unless and until you know a particular component is a bottleneck. I've had much better results optimizing globally first before optimizing locally. You can usually go back and optimize locally whenever you want, but optimizing globally is more often a one-shot deal as it typically requires non-local (i.e. cross-component) analysis and refactors, which nobody has time for.