Comment by acabal | Hacker News Reader

Comment by acabal | Hacker News Reader

utf8 vs utf8mb4 bit me in one of the most frustrating bugs I've ever had to debug.

At Scribophile members can write critiques for people's writing, with comments inserted inline. The underlying software dates back to the PHP5 days when MySQL only had the utf8 option. Everything was working fine for years, when all of a sudden users started complaining that from time to time, they would submit a critique and it would be mysteriously cut off at random places.

The problem was very intermittent, didn't happen very often, and there was no seeming rhyme or reason to it. But when it did happen, it was catastrophic because members would lose hours of work, seemingly at random!

All kinds of testing scaffolding and logging was put in place to try to find the problem with no luck. Then, after quite some time, we realized what the problem was.

At that time, emoji keyboards were brand new; most phones/tablets had limited support, and people didn't yet use phones and tablets for everything like they do now. But they were out there. Some users who were using these new emoji keyboards were inserting emoji smiley faces as they were writing critiques. In Scribophile's web interface, everything looked fine; but when the user submitted the critique, MySQL tried to insert a multibyte Unicode character into a regular utf8 field, and SILENTLY threw that character and all of the data after it away!!

Boy were we upset at MySQL about that one. But at least we figured it out!

I had the pleasure of finding that same behaviour in an application around 2015. When someone commented on a food order "Extra cheese please :folded hands: but no shrimp, I'm allergic."

But one thing I haven't found out is why some emojis did go through. The basic ones like a simple :smile: it seemed to me. Could it be that some only need 3 bytes and when more and more emojis got released, they went into 4 bytes?

Edit: HN stripped the emojis from my comment... I would put a rolling eyes emoji here if I could.

Exactly that - three byte emojis work fine in utf8 but 4 byte ones need utf8mb4 - and the four byte ones are the new ones that support the "color variations" and other similar things.

Apple adding those probably caused more upgrades to utf8mb4 than any amount of pleading from languages that actually needed them ever would have.

I looked through the master emoji list and I can't see that any expand into the 4th byte, but this stuff is REALLY complicated and I have no idea.

https://unicode.org/Public/emoji/14.0/emoji-test.txt

I believe you’re confusing two distinct concepts:

Sounds like you’re looking at how many scalar values are in an extended grapheme cluster (e.g. U+1F635 U+200D U+1F4AB dizzy face + zero width joiner + dizzy symbol, that’s one “character” made up of three scalar values).

The thing in question here is UTF-8 code units (bytes), and how many UTF-8 needs to encode a single scalar value. UTF-8 needs four bytes for anything above U+FFFF, and almost all emoji are above that.

I'm confused because U+1F635 can be stored in 3 bytes? I see nothing above 0xFFFFFF which would need the 4th byte? Or am I being dense?

In UTF-8, only some bits are used for the actual character codes. The rest are control bits for the decoder to know where a character starts and ends.

https://en.m.wikipedia.org/wiki/UTF-8#Encoding

U+1F635 is encoded in UTF-8 as bytes {f0, 9f, 98, b5}.

Thank you, that explains it!

To give a slightly more complete answer as to why, utf-8 was designed to be both backwards compatible with ASCII and self-synchronizing.

So you can't use half the values of a byte when encoding code points outside of ASCII, and the remaining 128 values need to be split between starting bytes and continuation bytes. At best you're going to fit about 6 bits into each byte, and unicode spans the equivalent of just over 20 bits, so you need 4 bytes max.

utf-8 opts to keep encoding very simple, and in 2-4 bytes it can store 11, 16, and 21 bits respectively.

If you discarded ASCII compatibility, you could create a 1-3 byte self-synchronizing unicode encoding. But at that point maybe you should be compressing your text instead.

Or you could weaken the compatibility and create "utf-1 but better" to fit into 3 bytes.

> Actually it does seem feasible to make an encoding that's backwards compatible with all printable ASCII characters, and uses 3 bytes for all other code points.

Sorry, you can’t squeeze Unicode in that way, even unsynchronised: you fall just over 2½ bits short. No, reserving the 95 printable ASCII characters scuttles your hopes, meaning you get less than 1½ bits of that byte to use for the rest of Unicode, when you needed a touch over 4 bits.

To represent all the rest: Unicode scalar values minus printable ASCII is U+0000–U+10FFFF minus U+D800–U+DFFF minus U+0020–U+007E: 1,111,969 possibilities, log₂(1111969) ≈ 20.085 bits of entropy. (Removing the 2048 surrogates and 95 printable ASCII characters barely made a dent, under 0.003 bits, but I had to remove it for informational correctness.)

95 printable ASCII characters, log₂(95) ≈ 6.57 bits. Subtract that reservation from the three bytes available and you’ve got about 17.43 bits left for the rest of Unicode, but you needed over 20.

[Edit: I observe you’ve removed what I was quoting before I posted my comment. Still fun mathematics.]

> If you discarded ASCII compatibility, you could create a 1-3 byte self-synchronizing unicode encoding.

Not quite. If you discarded ASCII compatibility, then you’d have almost four bits to spare, which is enough for a self-synchronising fixed-width 3-byte encoding (reserving the first bit of each byte to signal if it’s a start byte) or for an unsynchronised variable-width encoding of some kind; but to make it even 2-or-3 or 1-or-3 (rather than 1–3) bytes wide variable-width, you’d need four bits (3 + log₂ 2) for self-synchronisation, and you’re about 0.085 bits short. (1–3 bytes variable-width would require 3 + log₂ 3 ≈ 4.58 bits.)

—⁂—

I just got nerd-sniped, didn’t I?

I had to Google ⁂ as I've never seen that character before. Fascinating! It makes a nice Christmas snowflake stand-in.

I wonder what HN's rules are with sanitizing Unicode, as they won't allow emojis?

I use “—⁂—” as my horizontal rule in plain text, via my Compose key (Compose+h+r).

HN removes things that are officially marked as emoji, normalises whitespace to space (which annoys me because I sometimes want to type things like narrow no-break space in equations which it makes wider and loses the no-breakyness of), and that’s all I’ve noticed. Control characters probably get some treatment or other too.

> Sorry, you can’t squeeze Unicode in that way, even unsynchronised: you fall just over 2½ bits short. No, reserving the 95 printable ASCII characters scuttles your hopes, meaning you get less than 1½ bits of that byte to use for the rest of Unicode, when you needed a touch over 4 bits.

So it depends on exactly what we mean by compatible, and that's why I removed it.

But in particular, the encoding I had in mind when I wrote that was one that allows most ASCII characters as the middle byte of a triplet. That way whether you truncate from beginning or end you'll never have a rogue byte successfully decode.

So something like: Lead bytes from C0-FF, middle bytes from 00-BF, final bytes from 00-1F or 7F-BF.

If you also exclude null, tab, newline, and line feed, that gives you 64 x 188 x 93, which is just enough for every code point.

But arguably that's not compatible enough because you couldn't search for single ASCII characters with a dumb byte-wise algorithm.

> Not quite. If you discarded ASCII compatibility, then you’d have almost four bits to spare, which is enough for a self-synchronising fixed-width 3-byte encoding (reserving the first bit of each byte to signal if it’s a start byte) or for an unsynchronised variable-width encoding of some kind; but to make it even 2-or-3 or 1-or-3 (rather than 1–3) bytes wide variable-width, you’d need four bits (3 + log₂ 2) for self-synchronisation, and you’re about 0.085 bits short. (1–3 bytes variable-width would require 3 + log₂ 3 ≈ 4.58 bits.)

I don't know where you got that equation but it's not the right one for the situation.

Here's a really simple encoding just to disprove it: Leading bytes encode the plane (17 options) plus two more bits into decimal values 0 through 67. Continuation bytes encode 7 bits into decimal values 128 through 255. 17 planes + 2 bits + 7 bits + 7 bits, perfect fit. The remaining values, 68-127, can be used for single byte (and two byte) encodings.

But also the definition I was using for self-synchronizing doesn't seem to match the one on wikipedia. My intent was a code where you can guarantee sync after a specific small number of bytes (unlike UTF-1). In particular, if you let standalone bytes and continuation bytes overlap each other, you can still guarantee sync within 3 bytes. This gives you lots of space to play with, and you could make a very useful encoding. You could have >150 single byte codepoints and thousands of double byte codepoints and fit everything else into three bytes.

My (3 + log₂ 2) came from reckoning that you needed one whole bit as a signal in each byte (3), and that the byte-width thing (log₂ 2) was extra to that. As you correctly point out, this was incorrect: the first byte doesn’t need one full bit of entropy to signal that it’s the first byte.

If 21 bits seems like an odd place to stop, it’s because the Unicode consortium said they wouldn’t allocate any codepoints beyond U+10FFFF (the end of plane 16), because that’s as far as UTF-16 could go using a four-byte surrogate pair (10+10 bits, offset by U+10000).

Not that UTF-8 really stops at 21. As originally formulated it has 5 and 6 byte encodings for 26 and 31 bits, but an entire unsigned int of code points was later decided to be unnecessary, and those encodings were marked as invalid.

* Positive signed int, not unsigned, oops.

I've been struggling with fixing some edge cases with emojis in latin1 encoded DB this past couple of days and what you mention here is the most important thing I found yesterday (conincidence).

Unicode strings are made up of Unicode code points, but some code points combine with others to make graphemes (single visual characters). Some emojis, like flags (U+1F1EA U+1F1E6 ), are made of two code points.

EDIT: Flag unicode.

Correction: Unicode strings are made up of scalar values, not code points. The difference is that accursed mess that UTF-16 foist upon us, surrogates;oU+D800–U+DFFF are code points, but are not valid scalar values and cannot occur in Unicode strings (even if things like JavaScript are stupid in allowing potentially-malformed UTF-16, which is incompatible with Unicode, and Python similarly stupid in having strings that are not Unicode but are instead sequences of code points).

Thanks. I think I will need to read a proper book / article on this subject as encoding in general keeps haunting me periodically.

>Apple adding those probably caused more upgrades to utf8mb4 than any amount of pleading from languages that actually needed them ever would have.

I think there is a lesson here - maybe I'll find it after my coffee

You probably mean outside of the basic multilingual plane. Any non-ascii character is multibyte in utf8.

I hit the same issue in some app I wrote. It's what convinced me never to use MySQL ever again.

It's stories like this that stop me complaining online about a long day of futile debugging!

Reply on news.ycombinator.com