Ambiguous PNG Packer: Craft PNG files that appear different in Apple software
github.com
github.com
https://en.wikipedia.org/wiki/IBM_PCjr
https://en.wikipedia.org/wiki/IBM_Personal_Computer
(Am picking this nit because the PC is historically very important to our broad industry, and maybe people in the future should still be aware it was a very solidly-constructed piece of office equipment, not a relatively flimsy-looking home computer.)
It took us days to get to the root cause. Turns out that the issue wasn't Safari, it was Mac OS. Chrome was working fine because presumably they do the UTF-8 handling on their own.
I always hear rosy thing about how polished the Apple ecosystem is, but you can imagine how unamusing it is for my small business to spend $3000 of staff time on such a basic thing. I guess I understand it better now when people rant about Safari being the new IE.
It doesn’t help that I have a friend who works at Apple and acts as though the continued bugginess is my fault unless I take the time to undergo the pain of submitting bugs, for which one never receives a response and which are ignored for literally years on end. We are all unappreciated beta testers.
I've been trying to popularize "gamma testers" for years without much success.
Here's InfoWorld from 1994 - https://books.google.com/books?id=gzgEAAAAMBAJ&pg=PA59&dq=%2...
> Several readers used the same term for what they think the industry has made them become: gamma testers.
> "Gamma testing is just like beta testing in that you get a product that's too buggy for prime time and you help the vendor iron out the problems," explained one. "The only difference is that now we're bug testing what's supposed to be a shipping product that we paid for. And likely as not, these days they'll try to charge us for the privilege of reporting the bugs we find."
Oldest use I could find was from 1983, at https://archive.org/details/sim_micro-marketworld_1983-03-07... :
> Plus, WORDVISION is designed to be "support-free" after the sale, and our unique Pioneer's Club of "gamma testers" is helping insure we live up to that goal.
Here's a 2007 use describing a problem with the phrase, from https://archive.org/details/sdtimes184/page/n43/mode/2up?q=%...
> ... at least a decade since I started calling myself an unpaid gamma tester, for example. But people thought that one meant I specialized in displays and monitors...
What makes it worse is that you are PAYING to test their buggy software.
Apple hardware capable of running macOS may be expensive, but "locked down"? Definitely not. You can run Debian on anything from old PowerMacs to the new M1 model, although I admit that the driver situation on M1 isn't feature-complete.
Really? Please let me know where I can get the latest MacOS for free to replace Windows with it on my PC.
Besides, there’s guides-a-plenty all over the internet about making “Hackintoshes”.
By that logic iOS and Android are also free and you can alo get Windows for free with various OEM Laptops and system if we use that logic with the difference that you can choose to purchase Windows licenses to use on a machine you build yourself vs the inability to purchase an separate MacOS license without any Apple hardware.
So any comparisons on the price of the OS are moot since the cost of MacOS is priced into the accompanying hardware, MacBook, iMAc, etc as you cannot buy it without.
Microsoft also distributes Windows "for free" on their website and if you care not at all about license terms as we evidently don't when it comes to Mac OS, then it presents no moral quandary to disable its activation annoyances. Is Windows free?
Correction: It has no cost for Mac buyers. Those are the circumstances.
Violating the EULA that Apple makes you agree to during install (by making a Hackintosh), while a tortuous action, is not necessarily illegal, and it doesn't cost money.
Where please?
If that is the criteria for calling software free, then it's not a useful label.
Every legal macOS installation has been paid for with money.
It’s why I’d rather support opensource now. As, at least if no one cares to fix it, it can do it myself, or pay someone else to.
The bug: the default iOS mail client chokes if you copy and paste an email link into the to, cc or bcc field. Ie: “mail to:test@test.com”.
The response was “that’s how links work, that’s not a bug in the client”, and while technically that’s true. The best user experience would be parse out the “mailto:” when pasted into the to,cc,bcc field rather than failing to send the email.
And yes, from safari you can select “open in mail” and it will work. But if you need to copy a second mail link to cc, you’re stuffed.
And there’s no way to edit out the “mailto:” portion.
So the solution is to copy and paste the link into a text editor/notepad, remove the “mailto:” portion and then copy and paste the remaining portion into the mail client to,cc,bcc field.
And I think a better solution would be to either: A) automatically parse out the mailto: portion when pasting into the to,cc,bcc field.
Or B) prompt the user before doing A. Ie “do you mean to send to test@test.com?”
Not a single one was answered in the time I worked there. A few got some 'me toos'
At the same time I filed several bug reports for the new Gnome desktop that was rolling out and they all got answered, fixed or at least closed with a short answer.
It was only a short time I worked in that ecosystem. But as far as it worked out for me, I got zero support for several major bugs. Definitely not what I expected. Definitely not a OS I am willing to spend productive time on.
We did some unicode testing on different platforms and this was never an issue. Also to note these weren't complicated emojis or anything like that (e.g. character & skin-tone). Just usual characters with accents that are not in English.
The resolution for now has been using String.Prototype.normalize() with some tricks. It works for our case, but obviously it is not a silver bullet.
This article helped: https://medium.com/@sthadewald/the-utf-8-hell-of-mac-osx-fee...
Fix your expectations.
Some older filesystems, like HFS+ would normalize filenames. In APFS Apple decided not to normalize file names, but this caused a lot of issues. Later they have made APFS normalization-insensitive (so you cannot store two files with the same name, but different normalizations). Different normalizations are handled transparently by macOS frameworks.
On Linux with btrfs:
$ touch schön
$ cat $(echo -e "scho\u0308n")
cat: schön: No such file or directory
$ touch $(echo -e "scho\u0308n")
$ ls -l
total 0
-rw-r--r-- 1 daniel daniel 0 Dec 17 12:00 schön
-rw-r--r-- 1 daniel daniel 0 Dec 17 12:00 schön
$ ls | hexdump -c
0000000 s c h o 314 210 n \n s c h 303 266 n \n
Edit: updated to make this copy-pastable.
> So let's say you were looking up something based on the string, it was not there.
What this tells me is you're storing unicode text in a database without using unicode-aware string comparisons. This means that regardless of macOS, you could run into this issue just by having someone submit text in a manner that uses a different normalization form than whatever you've been using, your lookup will fail.
And it's not just NFC vs NFD. There's also NFKC and NFKD, which are the compatibility forms. For example, if I type fi (U+FB01 LATIN SMALL LIGATURE FI), that has the same NFC and NFD forms, but in NFKC and NFKD it becomes fi.
> This article helped: https://medium.com/@sthadewald/the-utf-8-hell-of-mac-osx-fee...
That article flips NFC and NFD.
> The resolution for now has been using String.Prototype.normalize() with some tricks.
Please fix your database to use a proper unicode-aware comparison instead. I don't know what database you're using but this may involve simply telling it what the text encoding of the column is. This is especially true when you start thinking about compatibility normalization (i.e. NFKC/NFKD). I believe the compatibility forms are generally appropriate for string comparisons (e.g. if someone searches for "fi" they should get results for "fi"; try it out in your browser's search field right now if you like) but you certainly don't want to store your text that way. And since you're presumably relying your database to do the lookup for you, this means you need your database to be unicode-aware, otherwise you have no choice but to store the compatibility canonicalization form in order to get search to be correct.
When you can be compatible with everything else, please try to be. MacOS isn't in this case. I would rather stand on the shoulder of giants who spare my staff from having to learn about "NFKC" and "NFKD". It's nicer to spend that time with our families. That's my humble opinion and personal values of course.
You say that like macOS is bucking the trend. NFC is a very common normalization form for text input, regardless of whether the browser chooses to normalize. And older versions of the W3C "Character Model for the World Wide Web: String Matching" note[1] even recommended using NFC explicitly, although the current version says that skipping Unicode normalization is the recommended form of matching for "new specifications".
As for Safari, it's been doing NFC normalization ever since 2006[2], and it sounds like this was originally at least partially-motivated by fixing a compatibility issue with Windows.
In any case, Safari converting text to NFC is certainly differing behavior from non-WebKit browsers, but if you give someone text in NFD and they delete and retype it, it's likely to end up in NFC anyway, which means this is a problem regardless of browser.
[1] https://www.w3.org/TR/charmod-norm/
[2] https://bugs.webkit.org/show_bug.cgi?id=8769
> I would rather stand on the shoulder of giants who spare my staff from having to learn about "NFKC" and "NFKD".
Then use Unicode-aware string comparison routines.
To render a page, we pulled up some JSON data from our database and rendered HTML based on that. To rehydrate the page, we baked the JSON data itself into the page too. We did that by naively JSON.stringify()'ing our database values into a script tag at the bottom of the page. The browser was essentially eval-ing the content in into a variable for us.
But oh how young and naive we were. Before long one of our clients started complaining that our web app was broken for them. We took a look and sure enough, the page was broken. We were scratching our heads for a few hours over that one. The client had somehow managed to insert a weird, invisible character into a string in one of the JSON objects we were rendering. It turns out, contrary to all expectations and sense, JSON isn't actually a strict subset of javascript. There are some perfectly valid JSON values which aren't valid javascript. Somehow our client had accidentally created one such value. When it was rendered back out into our script tag, the browser threw a parse error because our JSON wasn't valid javascript. And that made it abort loading our page's javascript bundle, and everything went sideways.
Of course, the other problem with this approach is that if any string happened to contain "</script>" then the browser would have considered that point the end of the javascript content. That would have also done weird, wild and potentially dangerous things to our page. But luckily we found that before any of our users stumbled on it. At least, as far as we know.
In fact, I just did try it out: searching for the ligature "fi" in Firefox (on Linux) does not find "fi". Neither does it work the other way around.
This suggests some questions.
1. Are fi.com and fi.com the same as far as DNS in concerned, or could both exist as separate websites?
2. If they are separate websites, do the various password manager browser extensions recognize they are different, or might they fill in your fi.com password when you are getting phished with fi.com? Same question for the various browser's built-in password and form savers.
It's extremely unhealthy for web standards that every deviation from Chrome's behavior is considered a nonsense bug now.
Using right-click context menus to copy/paste only works in Google Chrome. My understanding is that for security reasons, the browser doesn't allow websites to initiate a clipboard read/write. Google Slides recreates the right-click menu in JS, and so no clipboard events can be made from it. But Google Chrome apparently gives special permissions to Google websites, allowing them to have special enhanced privileges to initiate clipboard events.
I'd say that's definitely something that is an abuse of market share, as Google is using market dominance in the browser space to have a competitive advantage in the unrelated presentation software market.
Don't go down with that ship...
Also, just because Chrome feels faster and logs you into your google account without permission, it doesn't mean you shouldn't support alternative browsers. Do not only test your website in Chrome. It may get ignored as not working by non chrome users.
P.S.: I use Firefox, not Safari.
As who has spent a huge amount of time the last few weeks due to an extremely obscure and hard to reproduce Unicode bug in Windows, I can vouch for this. So can my ever receding hairline. Turns out, Unicode is hard.
this is why I use Qt and ship all the dependencies vetted manually on every platform (freetype, libav, etc). Only way to stay sane is to bypass the operating systems as much as possible and use the exact same code no matter the platform except for the barest-bone operations (opening a GL context, getting key/mouse events, etc). Text handling and rendering, video and image decoding, ... that should never depend on the OS you're using, those are properties of the apps. The more time passes the more I'm thinking that networking also should be deported into the apps too and bypass the OS network stack entirely given how much trouble it tends to give (hello the Bonjour / DNS-SD / ... mess).
I like having my fonts look the same everywhere, and only needing to configure them once. I also like my video and text do decode securely and accelerated.
I do, too ! On all the platforms I use ! (Windows, Mac and Linux). The only way to allow for that is to not use the OS rendering APIs.
> I also like my video and text do decode securely and accelerated.
And I trust ffmpeg much more for doing that correctly than whatever proprietary API there is in mac and windows.
I found that doing a small amount of activity like resizing other non-Safari windows caused extra glitchiness, so I was doing that throughout.
Apparently it's from the same author David Buchanan.
I don't think Apple have revealed the specifics of what process their human reviewers use (as a "security by obscurity" approach, and to prevent the bad press of admitting that they potentially subject low wage contractors to reviewing mentally damaging illegal images), but it's at least conceivable that an iPhone user could receive a series of images which look like (for example) bad scans of paper documents, but, when rendered on a reviewer's screen, look like images that require police attention.
I don’t have access to desktop Safari so I’m not sure what it’s meant to do.
Is there a tool that can detect ambiguous PNGs and other similar things?
https://superuser.com/questions/579216/why-does-this-png-ima...
Or with, uh, analog image processing:
https://en.wikipedia.org/wiki/The_dress
If you're hosting an image, you can also have some fun with referrer, user-agent, geo-ip, etc, based redirection. Where you post an image in a forum or similar, and different users see a different image.
Edit: It works on Safari (macbook pro), but not on my iphone safari.
Thankfully the bug is limited to how it's rendered...for now.