Hacker News Reader

Comment by a1371 | Hacker News Reader

This week we had a UTF-8 issue that only impacted Safari. One of my colleagues had bought a brand new Macbook Pro and our web-based software was fatally struggling with its default browser. Chrome was working fine.

It took us days to get to the root cause. Turns out that the issue wasn't Safari, it was Mac OS. Chrome was working fine because presumably they do the UTF-8 handling on their own.

I always hear rosy thing about how polished the Apple ecosystem is, but you can imagine how unamusing it is for my small business to spend $3000 of staff time on such a basic thing. I guess I understand it better now when people rant about Safari being the new IE.

View on HN

At the risk of repeating comments, welcome to Unicode Hell. And possibly to cross platform development :)

Also, just because Chrome feels faster and logs you into your google account without permission, it doesn't mean you shouldn't support alternative browsers. Do not only test your website in Chrome. It may get ignored as not working by non chrome users.

P.S.: I use Firefox, not Safari.

> At the risk of repeating comments, welcome to Unicode Hell. And possibly to cross platform development :)

As who has spent a huge amount of time the last few weeks due to an extremely obscure and hard to reproduce Unicode bug in Windows, I can vouch for this. So can my ever receding hairline. Turns out, Unicode is hard.

I had one a few weeks agos where, if the user had updated Safari, but hadn't updated macOS, then their browser didn't _actually_ support WebP, even though Safari 14 lists it as "supported".

File format support is baked into the OS. Apple actually makes static linking very difficult for exactly this reason--so they can reliably introduce support for new file formats on OS upgrade, rather than some apps needing to catch up.

The classic Safari nonsense bugs. When I see a new sentry issue popping up, I can instantly guess if it's going to be Safari or a real error just by looking at the error text.

And I could tell just from reading the comment that it was another case of developers not knowing about Unicode normalization.

It's extremely unhealthy for web standards that every deviation from Chrome's behavior is considered a nonsense bug now.

Take it as you want, I never have an issue with Firefox and it's not a Chrome based browser.

I've had a problem with Google Slides in Firefox, but I put the blame for that solely on Google.

Using right-click context menus to copy/paste only works in Google Chrome. My understanding is that for security reasons, the browser doesn't allow websites to initiate a clipboard read/write. Google Slides recreates the right-click menu in JS, and so no clipboard events can be made from it. But Google Chrome apparently gives special permissions to Google websites, allowing them to have special enhanced privileges to initiate clipboard events.

I'd say that's definitely something that is an abuse of market share, as Google is using market dominance in the browser space to have a competitive advantage in the unrelated presentation software market.

And I can tell an unprofessional web script kiddie based on whether they advocate browser monoculture.

I'm fine with any browser, as long as it's modern and up to the current standards (i.e not Safari)

Firefox can be a Chromium fork, too, but they get paid hundreds of millions to maintain a different engine by Google.

Don't go down with that ship...

> Chrome was working fine because presumably they do the UTF-8 handling on their own.

this is why I use Qt and ship all the dependencies vetted manually on every platform (freetype, libav, etc). Only way to stay sane is to bypass the operating systems as much as possible and use the exact same code no matter the platform except for the barest-bone operations (opening a GL context, getting key/mouse events, etc). Text handling and rendering, video and image decoding, ... that should never depend on the OS you're using, those are properties of the apps. The more time passes the more I'm thinking that networking also should be deported into the apps too and bypass the OS network stack entirely given how much trouble it tends to give (hello the Bonjour / DNS-SD / ... mess).

> Text handling and rendering, video and image decoding, ... that should never depend on the OS you're using

I like having my fonts look the same everywhere, and only needing to configure them once. I also like my video and text do decode securely and accelerated.

> I like having my fonts look the same everywhere,

I do, too ! On all the platforms I use ! (Windows, Mac and Linux). The only way to allow for that is to not use the OS rendering APIs.

> I also like my video and text do decode securely and accelerated.

And I trust ffmpeg much more for doing that correctly than whatever proprietary API there is in mac and windows.

What was the root cause and what was the fix?

We had unicode characters in strings in our database. When those were being fetched by the front-end on Safari, the browser was making them a different character. This meant if the exact same string was going back to our server after being in Safari, any kind of string-matching was failing. So let's say you were looking up something based on the string, it was not there.

We did some unicode testing on different platforms and this was never an issue. Also to note these weren't complicated emojis or anything like that (e.g. character & skin-tone). Just usual characters with accents that are not in English.

The resolution for now has been using String.Prototype.normalize() with some tricks. It works for our case, but obviously it is not a silver bullet.

This article helped: https://medium.com/@sthadewald/the-utf-8-hell-of-mac-osx-fee...

You said I always hear rosy thing about how polished the Apple ecosystem is. But this more sounds like a bug in your software. Many accented characters have two representations in Unicode, so you always have to normalize strings before making comparisons or use a comparison function that takes different encodings into account.

We learned that the hard way. Still, the basic dev expectation is to get the same thing out that they have put in. It shouldn't sound outrageous that we expected to get the same representation. That's all.

An alternative basic dev expectation is that given two different ways to represent the same thing, the software only gives you one canonical way.

Fix your expectations.

With all due respect, software development is complicated and it sounds like you just weren’t aware of this aspect of it. Which is fine. But it’s a bug in _your_ software, not Safari.

It sounds like this would be a nightmare in Unix filenames; even if everything in my system is perfectly clean utf-8, I could be trying to open "café.txt" and get a No such file or directory error because the input has é encoded a different way from what is saved on disk. You would need to list the directory contents, decode everything, perform the comparison in Unicode code points, and pray there is no more than one match. Any web UI that deals with files would have this kind of issues.

Yeah. Though it really depends on the system.

Some older filesystems, like HFS+ would normalize filenames. In APFS Apple decided not to normalize file names, but this caused a lot of issues. Later they have made APFS normalization-insensitive (so you cannot store two files with the same name, but different normalizations). Different normalizations are handled transparently by macOS frameworks.

On Linux with btrfs:

    $ touch schön
    $ cat $(echo -e "scho\u0308n")
    cat: schön: No such file or directory
    $ touch $(echo -e "scho\u0308n")
    $ ls -l
    total 0
    -rw-r--r-- 1 daniel daniel 0 Dec 17 12:00 schön
    -rw-r--r-- 1 daniel daniel 0 Dec 17 12:00 schön
    $ ls | hexdump -c
    0000000   s   c   h   o 314 210   n  \n   s   c   h 303 266   n  \n

Unicode, loads of fun :).

Edit: updated to make this copy-pastable.

I wonder if ZFS falls prey to the same issue. I've heard good things about it otherwise.

That's not a UTF-8 issue. That's Unicode. And macOS is not incorrect here.

> So let's say you were looking up something based on the string, it was not there.

What this tells me is you're storing unicode text in a database without using unicode-aware string comparisons. This means that regardless of macOS, you could run into this issue just by having someone submit text in a manner that uses a different normalization form than whatever you've been using, your lookup will fail.

And it's not just NFC vs NFD. There's also NFKC and NFKD, which are the compatibility forms. For example, if I type ﬁ (U+FB01 LATIN SMALL LIGATURE FI), that has the same NFC and NFD forms, but in NFKC and NFKD it becomes fi.

> This article helped: https://medium.com/@sthadewald/the-utf-8-hell-of-mac-osx-fee...

That article flips NFC and NFD.

> The resolution for now has been using String.Prototype.normalize() with some tricks.

Please fix your database to use a proper unicode-aware comparison instead. I don't know what database you're using but this may involve simply telling it what the text encoding of the column is. This is especially true when you start thinking about compatibility normalization (i.e. NFKC/NFKD). I believe the compatibility forms are generally appropriate for string comparisons (e.g. if someone searches for "ﬁ" they should get results for "fi"; try it out in your browser's search field right now if you like) but you certainly don't want to store your text that way. And since you're presumably relying your database to do the lookup for you, this means you need your database to be unicode-aware, otherwise you have no choice but to store the compatibility canonicalization form in order to get search to be correct.

Thanks for trying to be helpful. Of course, I over-simplified our 4 years of development and the specialized field we are in. There isn't really a database involved here. I am sure there are many more correct ways for us to do certain things, that said, I respectfully disagree and macOS is the issue here.

When you can be compatible with everything else, please try to be. MacOS isn't in this case. I would rather stand on the shoulder of giants who spare my staff from having to learn about "NFKC" and "NFKD". It's nicer to spend that time with our families. That's my humble opinion and personal values of course.

> When you can be compatible with everything else, please try to be. MacOS isn't in this case.

You say that like macOS is bucking the trend. NFC is a very common normalization form for text input, regardless of whether the browser chooses to normalize. And older versions of the W3C "Character Model for the World Wide Web: String Matching" note[1] even recommended using NFC explicitly, although the current version says that skipping Unicode normalization is the recommended form of matching for "new specifications".

As for Safari, it's been doing NFC normalization ever since 2006[2], and it sounds like this was originally at least partially-motivated by fixing a compatibility issue with Windows.

In any case, Safari converting text to NFC is certainly differing behavior from non-WebKit browsers, but if you give someone text in NFD and they delete and retype it, it's likely to end up in NFC anyway, which means this is a problem regardless of browser.

[1] https://www.w3.org/TR/charmod-norm/

[2] https://bugs.webkit.org/show_bug.cgi?id=8769

> I would rather stand on the shoulder of giants who spare my staff from having to learn about "NFKC" and "NFKD".

Then use Unicode-aware string comparison routines.

Heh... This reminds me of a bug we ran into a few years ago when rehydrating JSON content for a website my team was working on.

To render a page, we pulled up some JSON data from our database and rendered HTML based on that. To rehydrate the page, we baked the JSON data itself into the page too. We did that by naively JSON.stringify()'ing our database values into a script tag at the bottom of the page. The browser was essentially eval-ing the content in into a variable for us.

But oh how young and naive we were. Before long one of our clients started complaining that our web app was broken for them. We took a look and sure enough, the page was broken. We were scratching our heads for a few hours over that one. The client had somehow managed to insert a weird, invisible character into a string in one of the JSON objects we were rendering. It turns out, contrary to all expectations and sense, JSON isn't actually a strict subset of javascript. There are some perfectly valid JSON values which aren't valid javascript. Somehow our client had accidentally created one such value. When it was rendered back out into our script tag, the browser threw a parse error because our JSON wasn't valid javascript. And that made it abort loading our page's javascript bundle, and everything went sideways.

Of course, the other problem with this approach is that if any string happened to contain "</script>" then the browser would have considered that point the end of the javascript content. That would have also done weird, wild and potentially dangerous things to our page. But luckily we found that before any of our users stumbled on it. At least, as far as we know.

The worse one is opening another script tag right after closing the first. XSS dream.

I've noticed on several occasions that text written by Mac users and published online would replace the Swedish letters ä and ö with a¨ and o¨. I think it's due to some combination of how they do Unicode normalization and how Firefox on Windows renders the text. I haven't seen it for at least a year, so I'm guessing it's resolved now.

> if someone searches for "ﬁ" they should get results for "fi"; try it out in your browser's search field right now if you like

In fact, I just did try it out: searching for the ligature "ﬁ" in Firefox (on Linux) does not find "fi". Neither does it work the other way around.

Same with Firefox on MacOS. Chrome and Safari do find "fi" for "ﬁ" and "ﬁ" for "fi".

This suggests some questions.

1. Are ﬁ.com and fi.com the same as far as DNS in concerned, or could both exist as separate websites?

2. If they are separate websites, do the various password manager browser extensions recognize they are different, or might they fill in your fi.com password when you are getting phished with ﬁ.com? Same question for the various browser's built-in password and form savers.

https://en.wikipedia.org/wiki/Punycode

Never compare unicode strings on the binary level. Or if you do, convert both sides to same same form, which you basically did in the end.

Safari still defaults to something other than utf8.

Yes, and this threw us off because we thought we weren't communicating the encoding correctly with Safari. A day later we finally realized that's not the issue.

So does javascript, in all web browsers and environments.

JavaScript’s DOMString is basically defined as UTF-16.

To be fair it reads to me a Unicode problem rather than macOS.

From my perspective, the hours worked may add up to $3000 in salary, but the staff member who solved the problem may feel very good about their accomplishment.

Unfortunately this issue made them not have a demo ready by the anticipated date, so the good feeling didn't really prevail.

Apple has severely cut back on QA and simply cannot claim to publish quality software anymore. The number of very easy to find bugs I run into makes that obvious. As an example, the latest iOS update introduced a bug that causes the input box to disappear completely on an iMessage thread so I can’t send messages unless I close it and come back. Happens daily. Embarrassing.

It doesn’t help that I have a friend who works at Apple and acts as though the continued bugginess is my fault unless I take the time to undergo the pain of submitting bugs, for which one never receives a response and which are ignored for literally years on end. We are all unappreciated beta testers.

> We are all unappreciated beta testers.

I've been trying to popularize "gamma testers" for years without much success.

So have others!

Here's InfoWorld from 1994 - https://books.google.com/books?id=gzgEAAAAMBAJ&pg=PA59&dq=%2...

> Several readers used the same term for what they think the industry has made them become: gamma testers.

> "Gamma testing is just like beta testing in that you get a product that's too buggy for prime time and you help the vendor iron out the problems," explained one. "The only difference is that now we're bug testing what's supposed to be a shipping product that we paid for. And likely as not, these days they'll try to charge us for the privilege of reporting the bugs we find."

Oldest use I could find was from 1983, at https://archive.org/details/sim_micro-marketworld_1983-03-07... :

> Plus, WORDVISION is designed to be "support-free" after the sale, and our unique Pioneer's Club of "gamma testers" is helping insure we live up to that goal.

Here's a 2007 use describing a problem with the phrase, from https://archive.org/details/sdtimes184/page/n43/mode/2up?q=%...

> ... at least a decade since I started calling myself an unpaid gamma tester, for example. But people thought that one meant I specialized in displays and monitors...

> We are all unappreciated beta testers.

What makes it worse is that you are PAYING to test their buggy software.

Unlike Microsoft, Apple doesn't make anyone pay for the OS since... 2013 IIRC?

Don't you just get a free license when you buy their expensive ~locked down~ hardware? Afaik you don't get free osx licenses for hackintosh or virtual build machines.

> Don't you just get a free license when you buy their expensive locked down hardware?

Apple hardware capable of running macOS may be expensive, but "locked down"? Definitely not. You can run Debian on anything from old PowerMacs to the new M1 model, although I admit that the driver situation on M1 isn't feature-complete.

You are right, locked down is definitely to strong. My assumption is based on driver issues you might face in the past.

On the other hand iPhones and iPads are very locked down.

>Unlike Microsoft, Apple doesn't make anyone pay for the OS since... 2013 IIRC?

Really? Please let me know where I can get the latest MacOS for free to replace Windows with it on my PC.

Don’t be like that. Providing it for free does not mean that it has to work on your device. By that definition, no software (including non-Apple) for macOS is ever free because it requires a more expensive computer to run.

Besides, there’s guides-a-plenty all over the internet about making “Hackintoshes”.

Then the comparison on pricing between Windows and MacOS that grandparent made doesn't make sense if one cannot be bough without accompanying hardware, unlike the other, effectively making MacOS priced into the hardware itself, so the comparison falls flat from the start.

By that logic iOS and Android are also free and you can alo get Windows for free with various OEM Laptops and system if we use that logic with the difference that you can choose to purchase Windows licenses to use on a machine you build yourself vs the inability to purchase an separate MacOS license without any Apple hardware.

So any comparisons on the price of the OS are moot since the cost of MacOS is priced into the accompanying hardware, MacBook, iMAc, etc as you cannot buy it without.

Characterizing macOS as something Apple intends to distribute as free is mostly false. They intend to charge for it with the purchase of a Mac.

Microsoft also distributes Windows "for free" on their website and if you care not at all about license terms as we evidently don't when it comes to Mac OS, then it presents no moral quandary to disable its activation annoyances. Is Windows free?

There is no license that can be purchased for MacOS. It literally has no cost in any circumstance. This is just tortured sophistry to arrive at the tremendous conclusion that Windows does, and sometimes doesn’t, cost money. Is windows free? "Sometimes!"

>It literally has no cost in any circumstance.

Correction: It has no cost for Mac buyers. Those are the circumstances.

You're intentionally stepping around my arguments. Sure, it has no cost for Mac owners, but the ISO is free to download, and Hackintoshes are very much a thing. Again: if we use your definition, then anything requiring paid hardware cannot, by definition, be free. Which is absurd to suggest. No one I've met claims that free macOS exclusive software isn't actually free. Why is the OS any different.

Violating the EULA that Apple makes you agree to during install (by making a Hackintosh), while a tortuous action, is not necessarily illegal, and it doesn't cost money.

>but the [MacOS] ISO is free to download

Where please?

https://github.com/corpnewt/gibMacOS

It's not tortured, it's strictly the only supported and legal way of running it. Anything else is software piracy.

If that is the criteria for calling software free, then it's not a useful label.

Every legal macOS installation has been paid for with money.

When was the last time you heard of someone actually buying a copy of Windows, though? I've owned a lot of different versions of Windows in the past 30 years, but never paid for a single one.

I know someone who bought a copy of Windows 10 for their computer, because they asked me to help install it. It was definitely not an OEM thing, and the packaging had a USB drive in it.

Yea, I’m feeling this. I submitted a bug report, and while there was a lot of “care”, the result was “not a bug, won’t fix”. On the one hand, I can understand that. But on the other, it’s a crappy user experience, and the apple I supported, I’d expect to fix it.

It’s why I’d rather support opensource now. As, at least if no one cares to fix it, it can do it myself, or pay someone else to.

The bug: the default iOS mail client chokes if you copy and paste an email link into the to, cc or bcc field. Ie: “mail to:test@test.com”.

The response was “that’s how links work, that’s not a bug in the client”, and while technically that’s true. The best user experience would be parse out the “mailto:” when pasted into the to,cc,bcc field rather than failing to send the email.

And yes, from safari you can select “open in mail” and it will work. But if you need to copy a second mail link to cc, you’re stuffed.

Copy'n'pasting "mailto:test@test.com" into gmail's new mail "to:" field also doesn't magically remove the mailto: part either.

Ok. Well I’d consider them both subpar experiences. And I think it quite clearly could be better.

Chokes as in, it freezes up or something?

As in, it doesn’t send the email.

And there’s no way to edit out the “mailto:” portion.

So the solution is to copy and paste the link into a text editor/notepad, remove the “mailto:” portion and then copy and paste the remaining portion into the mail client to,cc,bcc field.

And I think a better solution would be to either: A) automatically parse out the mailto: portion when pasting into the to,cc,bcc field.

Or B) prompt the user before doing A. Ie “do you mean to send to test@test.com?”

I had a Macbook pro for about 6 months about 5 years ago. I filed afaik 5 bugs, 4 of them for screen related issues. (no external screen without powering the laptop, external screens don't come back after sleep, etc ..) all bugs that have been in the whole series (other colleagues got the same 2500$ beast, with the same issues)

Not a single one was answered in the time I worked there. A few got some 'me toos'

At the same time I filed several bug reports for the new Gnome desktop that was rolling out and they all got answered, fixed or at least closed with a short answer.

It was only a short time I worked in that ecosystem. But as far as it worked out for me, I got zero support for several major bugs. Definitely not what I expected. Definitely not a OS I am willing to spend productive time on.

Reply on news.ycombinator.com