Hacker News Reader

Comment by a1371 | Hacker News Reader

We had unicode characters in strings in our database. When those were being fetched by the front-end on Safari, the browser was making them a different character. This meant if the exact same string was going back to our server after being in Safari, any kind of string-matching was failing. So let's say you were looking up something based on the string, it was not there.

We did some unicode testing on different platforms and this was never an issue. Also to note these weren't complicated emojis or anything like that (e.g. character & skin-tone). Just usual characters with accents that are not in English.

The resolution for now has been using String.Prototype.normalize() with some tricks. It works for our case, but obviously it is not a silver bullet.

This article helped: https://medium.com/@sthadewald/the-utf-8-hell-of-mac-osx-fee...

View on HN

That's not a UTF-8 issue. That's Unicode. And macOS is not incorrect here.

> So let's say you were looking up something based on the string, it was not there.

What this tells me is you're storing unicode text in a database without using unicode-aware string comparisons. This means that regardless of macOS, you could run into this issue just by having someone submit text in a manner that uses a different normalization form than whatever you've been using, your lookup will fail.

And it's not just NFC vs NFD. There's also NFKC and NFKD, which are the compatibility forms. For example, if I type ﬁ (U+FB01 LATIN SMALL LIGATURE FI), that has the same NFC and NFD forms, but in NFKC and NFKD it becomes fi.

> This article helped: https://medium.com/@sthadewald/the-utf-8-hell-of-mac-osx-fee...

That article flips NFC and NFD.

> The resolution for now has been using String.Prototype.normalize() with some tricks.

Please fix your database to use a proper unicode-aware comparison instead. I don't know what database you're using but this may involve simply telling it what the text encoding of the column is. This is especially true when you start thinking about compatibility normalization (i.e. NFKC/NFKD). I believe the compatibility forms are generally appropriate for string comparisons (e.g. if someone searches for "ﬁ" they should get results for "fi"; try it out in your browser's search field right now if you like) but you certainly don't want to store your text that way. And since you're presumably relying your database to do the lookup for you, this means you need your database to be unicode-aware, otherwise you have no choice but to store the compatibility canonicalization form in order to get search to be correct.

Thanks for trying to be helpful. Of course, I over-simplified our 4 years of development and the specialized field we are in. There isn't really a database involved here. I am sure there are many more correct ways for us to do certain things, that said, I respectfully disagree and macOS is the issue here.

When you can be compatible with everything else, please try to be. MacOS isn't in this case. I would rather stand on the shoulder of giants who spare my staff from having to learn about "NFKC" and "NFKD". It's nicer to spend that time with our families. That's my humble opinion and personal values of course.

> When you can be compatible with everything else, please try to be. MacOS isn't in this case.

You say that like macOS is bucking the trend. NFC is a very common normalization form for text input, regardless of whether the browser chooses to normalize. And older versions of the W3C "Character Model for the World Wide Web: String Matching" note[1] even recommended using NFC explicitly, although the current version says that skipping Unicode normalization is the recommended form of matching for "new specifications".

As for Safari, it's been doing NFC normalization ever since 2006[2], and it sounds like this was originally at least partially-motivated by fixing a compatibility issue with Windows.

In any case, Safari converting text to NFC is certainly differing behavior from non-WebKit browsers, but if you give someone text in NFD and they delete and retype it, it's likely to end up in NFC anyway, which means this is a problem regardless of browser.

[1] https://www.w3.org/TR/charmod-norm/

[2] https://bugs.webkit.org/show_bug.cgi?id=8769

> I would rather stand on the shoulder of giants who spare my staff from having to learn about "NFKC" and "NFKD".

Then use Unicode-aware string comparison routines.

Heh... This reminds me of a bug we ran into a few years ago when rehydrating JSON content for a website my team was working on.

To render a page, we pulled up some JSON data from our database and rendered HTML based on that. To rehydrate the page, we baked the JSON data itself into the page too. We did that by naively JSON.stringify()'ing our database values into a script tag at the bottom of the page. The browser was essentially eval-ing the content in into a variable for us.

But oh how young and naive we were. Before long one of our clients started complaining that our web app was broken for them. We took a look and sure enough, the page was broken. We were scratching our heads for a few hours over that one. The client had somehow managed to insert a weird, invisible character into a string in one of the JSON objects we were rendering. It turns out, contrary to all expectations and sense, JSON isn't actually a strict subset of javascript. There are some perfectly valid JSON values which aren't valid javascript. Somehow our client had accidentally created one such value. When it was rendered back out into our script tag, the browser threw a parse error because our JSON wasn't valid javascript. And that made it abort loading our page's javascript bundle, and everything went sideways.

Of course, the other problem with this approach is that if any string happened to contain "</script>" then the browser would have considered that point the end of the javascript content. That would have also done weird, wild and potentially dangerous things to our page. But luckily we found that before any of our users stumbled on it. At least, as far as we know.

The worse one is opening another script tag right after closing the first. XSS dream.

I've noticed on several occasions that text written by Mac users and published online would replace the Swedish letters ä and ö with a¨ and o¨. I think it's due to some combination of how they do Unicode normalization and how Firefox on Windows renders the text. I haven't seen it for at least a year, so I'm guessing it's resolved now.

> if someone searches for "ﬁ" they should get results for "fi"; try it out in your browser's search field right now if you like

In fact, I just did try it out: searching for the ligature "ﬁ" in Firefox (on Linux) does not find "fi". Neither does it work the other way around.

Same with Firefox on MacOS. Chrome and Safari do find "fi" for "ﬁ" and "ﬁ" for "fi".

This suggests some questions.

1. Are ﬁ.com and fi.com the same as far as DNS in concerned, or could both exist as separate websites?

2. If they are separate websites, do the various password manager browser extensions recognize they are different, or might they fill in your fi.com password when you are getting phished with ﬁ.com? Same question for the various browser's built-in password and form savers.

https://en.wikipedia.org/wiki/Punycode

You said I always hear rosy thing about how polished the Apple ecosystem is. But this more sounds like a bug in your software. Many accented characters have two representations in Unicode, so you always have to normalize strings before making comparisons or use a comparison function that takes different encodings into account.

We learned that the hard way. Still, the basic dev expectation is to get the same thing out that they have put in. It shouldn't sound outrageous that we expected to get the same representation. That's all.

An alternative basic dev expectation is that given two different ways to represent the same thing, the software only gives you one canonical way.

Fix your expectations.

With all due respect, software development is complicated and it sounds like you just weren’t aware of this aspect of it. Which is fine. But it’s a bug in _your_ software, not Safari.

It sounds like this would be a nightmare in Unix filenames; even if everything in my system is perfectly clean utf-8, I could be trying to open "café.txt" and get a No such file or directory error because the input has é encoded a different way from what is saved on disk. You would need to list the directory contents, decode everything, perform the comparison in Unicode code points, and pray there is no more than one match. Any web UI that deals with files would have this kind of issues.

Yeah. Though it really depends on the system.

Some older filesystems, like HFS+ would normalize filenames. In APFS Apple decided not to normalize file names, but this caused a lot of issues. Later they have made APFS normalization-insensitive (so you cannot store two files with the same name, but different normalizations). Different normalizations are handled transparently by macOS frameworks.

On Linux with btrfs:

    $ touch schön
    $ cat $(echo -e "scho\u0308n")
    cat: schön: No such file or directory
    $ touch $(echo -e "scho\u0308n")
    $ ls -l
    total 0
    -rw-r--r-- 1 daniel daniel 0 Dec 17 12:00 schön
    -rw-r--r-- 1 daniel daniel 0 Dec 17 12:00 schön
    $ ls | hexdump -c
    0000000   s   c   h   o 314 210   n  \n   s   c   h 303 266   n  \n

Unicode, loads of fun :).

Edit: updated to make this copy-pastable.

I wonder if ZFS falls prey to the same issue. I've heard good things about it otherwise.

Never compare unicode strings on the binary level. Or if you do, convert both sides to same same form, which you basically did in the end.

To be fair it reads to me a Unicode problem rather than macOS.

Safari still defaults to something other than utf8.

Yes, and this threw us off because we thought we weren't communicating the encoding correctly with Safari. A day later we finally realized that's not the issue.

So does javascript, in all web browsers and environments.

JavaScript’s DOMString is basically defined as UTF-16.

Reply on news.ycombinator.com