How many replacement characters?

A well laid out case. Given the churn the new change in preference it might cause in the ecosystem, it's probably worth reverting, especially since apparently no real benefits beyond "it feels right" (according to this accounting) have been put forth.

It may cause the committee to lose a little face, but less than digging in your heels over a decision that has no ramifications to you but does to others, and you have no real justification for. Hopefully they see that and acquiesce, or at least come back with a well thought out rebuttal that isn't dismissive.

I completely agree with the author on this. The old Unicode 9.0 best practice made sense for a UTF-8 decoder that consumed input using a state machine, and like the article says, if you accept the correct byte-ranges in each state as in Table 3-7, your state-machine-based decoder will implicitly reject every kind of invalid sequence, including overlong encodings, encoded UTF-16 surrogates and encoded out of range characters. A state machine also makes other things trivial, like validating UTF-8 input without outputting characters and writing a streaming UTF-8 to UTF-16 converter. Another property is that it rejects invalid input as soon as possible, eg. as soon as it consumes input that can't possibly be part of an invalid sequence.

This is an example of a particularly elegant UTF-8 decoder which uses a state machine: http://bjoern.hoehrmann.de/utf-8/decoder/dfa/

The proposed Unicode 11 best practice suits a decoder that checks for overlong sequences, encoded surrogates and out of range values as a sort of post-validation step, after consuming all the bytes of a potentially valid UTF-8 sequence. Not only is this different to the behaviour of every existing decoder except for ICU, it also seems less elegant, more complicated and more error prone to me. If I understand correctly, even bytes like 0xc0 and 0xf5, which never form part of a valid UTF-8 sequence, won't be rejected immediately in this kind of decoder.

The article makes a pretty solid argument for why this difference in behaviour matters, even though it's just a best practice and not an official requirement. The two key points in this for me are that most existing UTF-8 decoders produce identical results matching the Unicode 9.0 best practice, and that there was an actual bug in Chrome when two internal UTF-8 decoders produced differing results. I think I'd make a stronger conclusion though: Not only should they keep the current recommended best practice, they should also elevate it to a requirement in order to prevent bugs like the one in Chrome from happening in future. Most existing UTF-8 decoders are already compliant, and it would be a nice property of UTF-8 if all byte sequences, including invalid ones, decoded to the same sequence of codepoints in every decoder.

How I have implemented things is that when the UTF-8 decoder encounters an invalid sequence, it retreats to the beginning of that sequence. It then converts the first byte of that sequence to a replacement characters, and consumes it. Then it resets to its initial state and begins decoding starting at the following byte.

Basically we are saying "no match occurs for a valid UTF-8 pattern at this input position; let's do error recovery by dropping a byte, pooping it out as a replacement character and trying again."

I have used the surrogate pair range U+DC00 - U+DCFF for replacement characters. On output, I convert these back to individual bytes. Thus the end-to-end decode+encode is binary transparent: any byte string can be decoded to a sequence of code points, some of which may be replacement characters, and that sequence will encode back to the original byte string. (This requirement cannot be achieved if multiple bogus bytes are collapsed into one replacement character.)

Well, that's not the full story: to have this transparency property, we also need to ensure that when some U+DCXX occurs by means of a valid UTF-8 pattern, we nevertheless treat it as invalid. I.e. there is a rule that if the UTF-8 decode works, but a U+DCXX code-point emerges, then we retreat to the start and drop a byte as a replacement character, as if a bad code had been seen.

The author found it hard to "find the right API entry point in Go documentation".

For the record, Go produces one U+FFFD per byte, not per maximal contiguous run, when iterating over bad UTF-8. This is part of the language specification, not just a library, although the standard libraries follow this behavior. For example, in the standard UTF-8 library, https://golang.org/pkg/unicode/utf8/#DecodeRune says that the size returned is 1 (i.e. 1 byte) for invalid UTF-8.

The relevant language spec section is https://golang.org/ref/spec#For_statements and look for "If the iteration encounters an invalid UTF-8 sequence, the second value will be 0xFFFD, the Unicode replacement character, and the next iteration will advance a single byte in the string."

Example code: https://play.golang.org/p/OLIWcjLIvF

I'll note that both Go and UTF-8 were invented by Ken Thompson and Rob Pike. I'm sure that the Go authors were aware of UTF-8's details. (Go also involved Robert Griesemer, but that's tangential).

Thought: what if Unicode decoding was "lossless" in the face of errors, such that the replacement characters represented the bitstring of non-decodable bytes? (E.g. 256 reserved-codepoints for each possible octet value, that render as e.g. "[FF]" in a box; and then another 255 for the set of 7-bit, 6-bit, 5-bit, etc. overhangs.)

> The proposal is ambiguous about whether to do the same thing for five and six-byte sequences whose bit pattern is not defined as existing in Unicode but was defined in now-obsolete RFCs for UTF-8 [...] If five and six-byte sequences are treated according to the logic of the newly-accepted proposal, the newly-accepted proposal matches the behavior of ICU.

Regarding 5- and 6-byte sequences, perhaps the Unicode Consortium in their ambiguity and the ICU in its implementation are allowing for their possible return to Unicode. One day in the far-off future when UTF-16 finally dies, it will be feasible to increase the codepoint repetoire back up from 1 million to 2 billion, which is easy to implement in both UTF-8 and UTF-16.

This raises what appear to be important points. Has it been formally submitted to Unicode in some way?

It seems like only harm is that "implementations have to explain themselves" and one Chromium bug.