Has goto fail been fixed yet? | Hacker News Reader

Has goto fail been fixed yet?

hasgotofailbeenfixedyet.com

106 pointselectic96 comments

This is absurd and unconscionable. There is no excuse whatsoever for not having a 10.9 patch ready to go at the same time as iOS. There is especially no excuse for still not having a 10.9 patch five days later.

I would love to see a detailed postmortem about exactly how this bug happened in the first place and why it's taking so long to fix it on the Mac side. Unfortunately, given how secretive the company is, I'm sure we'll never have more than speculation.

I wonder if they are finding that a lot of their systems inadvertently relied upon this bug, and they are scrambling to test and fix them all rather than release a fix that causes a lot of other things to break

Wonder if we'll see a patch for this 0day this side of WWDC...

More likely they have to regression test a whole ton of things, just in case, and do all the recompiling in correct order. I can't see anything actually depending on the bug, but making sure they don't screw up the patch is hard.

But a lot of programmers who have never done anything more difficult than mylamesocialstartup.com in PHP have no idea what it's like to build and test something as complex as an OS. No, recompiling your Linux kernel ain't the same thing.

If they need more than a day to recompile one library with a one-line change and see if it still boots and runs software update, then they have much bigger issues than this single bug.

It's now been 5 days PLUS however long they sat on this for ios.

They do have a bigger issue. The bigger issue is that they're releasing an entire operating system, with 15 years or so of accumulated cruft and process. And it's not just a one line change... they need to understand the implications of the bug across their system and create regression tests to catch all those corner cases, then do the "one line change" and properly regression test it everywhere.

Again, this isn't some little web site. This is a piece of software that thousands of programmers have worked on for more than a decade, millions of lines of code with a tremendously complex build, testing, release, and approval system.

And finally, Apple's primary concern isn't that a handful of customers might be exposed to risk for a couple of extra days. It's that they botch the rollout of a major security patch and have to re-release, or customers are victimized by new bugs as part of the patch. That sort of thing can cost them billions in market punishment.

What? It's a one-line, clearly understood bugfix. (The extra goto wasn't there in 10.8.x and that hasn't caused any problems) Rebuild the library, spend a few days regression testing at least software update so you can re-push another update, and ship it at the same time as the ios and appletv fixes!

You don't spend a week holding the fix hostage so you can fine tune a new os release with "improvements to autofill forms" in safari while script kiddies are running wild with mitmproxy.

Also I would think that a full ios+appletv OS release is much more complicated than an OSX release due to the way the mobile OSes are packaged (probably 10+ unique/model restore images plus delta downloads)

It's not like iOS is some tiny little thing, but they got it out much sooner.

If their build process is so broken that it takes days to take the 10.9.1 tag, apply a one-line patch, and release it, then they're doing it severely wrong. Security problems happen, and you need to be ready to move fast.

Personally, if I was on that team and it came down to taking days to recompile and retest everything, I'd be seriously considering a binary patch as an interim fix. Take the actual built binaries, apply this one patch, and you know nothing else got somehow miscompiled or mislinked in the process.

"No, recompiling your Linux kernel ain't the same thing."

You're right. It's not at all like Red Hat compiling and releasing a new RPM which they would push out in hours, max. It's actually much easier. Apple retains absolute control over their source code.

Most likely it's that they're trying to do the patch in 10.9.2, and have a lot of non-bug-fix related 10.9.2 stuff to work out (they've done 7 prerelease builds to developers of 10.9.2 already).

The manager who is holding the SSL fix hostage to 10.9.2's shitty feature-adds should be terminated out of a cannon. It's hilarious that we don't know who that person is -- who is the head of software security at Apple? From what I've heard there are at least two separate teams, and the relevant team is closely tied to Federighi, but no one outside Apple (generally) knows sub-SVP people at Apple.

There should be 10.9.1.1 with just the SSL fix. If that means delaying 10.9.2 another month, so be it.

I'm a little naive when it comes to SSL/TLS. I've been wondering whether the reason for the delay is that because with this compromise Apple's update service is no longer a secure channel through which to distribute a fix. So now they're scratching their heads trying to figure out a way around the issue, possibly coding up something that uses OpenSSL. Is this line of reasoning unfounded?

EDIT: Great points about the checking of the signatures. Let's hope there's not a second bug that can bypass this in some cases.

My understanding is that the update system checks digital signatures on the downloaded data separately from TLS, rather than simply relying on the integrity of TLS. If that understanding is correct, then there shouldn't be any issue there.

Software updates are signed by Apple's key, only signed and verified files can be installed by software update

That's an interesting line of thought, but:

1. if they can't use the update service, then they can't deploy anything that they're coding up anyways.

2. I assume they sign their updates, so it shouldn't matter that the channel is compromised.

That would be all the more reason for them to have released the update simultaneously with the iOS update.

Although we still don't know how this was discovered or if it's in the wild.

I am also very surprised by the lack of outrage in media, either mainstream or even tech related (including blogs and HN / netsec).

It's definitely absurd, but come on, unconscionable makes it sound like it's more irresponsible than the industry standard - when in fact it is less.

My guess is that they are close to shipping 10.9.2 and decided to hold on for that, not realizing how much attention this bug would get.

Yes, I agree that's a stupid call to make, but most smartphones have far worse unpatched vulnerabilities.

Also, it's worth saying that negative attention probably has little effect on Apple. In this case it's totally justified and well deserved, but the vast majority of it is 'they rejected my flappy bird app', or 'why Apple should fire Tim cook', so this doesn't add much to the pile.

Do you have examples of these far worse unpatched vulnerabilities? Not saying they aren't there, but I'd like to know what they are.

Anyway, I stand by "unconscionable" either way. It implies nothing about the industry standard. If Apple's unconscionable actions here are still better than the industry standard (and I'm skeptical about that, hence the "if", but even given that) then that just means the industry standard is unconscionable.

Why is it worth saying that negative attention probably had little effect on Apple? I really could not possibly care less about this. My statements are based on impact to the users including myself, not the price of AAPL or its balance sheet.

Apparently, there ARE some things $100M in cash can't buy, like security reviewers, security teams, security coders.. hmm, actually, $100M in cash could buy all those things. Guess it's a question of priorities, and security isn't one of them.

You mean $100B in cash.

Yes - it's obvious that there are bad practices at work here - in particular no strict static analysis. I agree there's little excuse for not adding machine processes that could have helped.

But adding human processes costs time and agility, and as you point out, money cannot replace these.

British million = American billion = 10^9

MM is used to denote an American million = 10^6

Are you sure? The old 'British Billion' was 10^12, but even that is rarely used any more. I've never seen an 'M' suffix used to mean anything other than 10^6.

M is widely used as "thousand" in specialized contexts like "CPM" and "RPM" within the advertising/accounting space.

I don't think that's right. 10^6 is a million in both languages.

10^9 used to be a milliard in British English but is no longer used. It is now referred to as a billion.

10^12 used to be a billion in British English but is now referred to as a trillion.

Edit: http://www.youtube.com/watch?v=C-52AI_ojyQ explains it all quite nicely.

Even if they spent every last cent of that money on security, they would still have bugs. You never get rid of all of them.

People that talk about security spending miss the inherent imbalance in security: a defender has to find every bug; an attacker only has to find one.

That doesn't mean you can't make it harder for the attacker.

Nobody is saying that Apple's wealth should mean they have no bugs. However, it does mean that they should have reasonable test coverage of critical security code.

Or even if not test coverage - static analysis.

Here's an example: http://m.slashdot.org/story/198449

And these android vulnerabilities are essentially unpatchable by design - which I think is something that really does deserve the word unconscionable.

But I accept that the overall behavior in the industry could be described by that term.

My comment about the negative attention is that Apple makes decisions based on the impact to itself. A few days of delay in fixing the bug might result in some small number of compromises, but as long as it doesn't drag on for months, it's unlikely to be a big deal.

On the other hand, negative attention and social media sentiment could have been a big deal for Apple, had it not been for all the wolf-crying. Now it's just a muted signal.

So from Apple's position this is actually a minor affair. We are horrified in principle but they are basing their actions on their practical reality.

Sheesh. I retract my "skeptical about that" bit.

I think you're right that this won't be a big deal for Apple. It won't really hurt them much.

However, basing your actions solely on the impact to yourself and ignoring the potentially huge negative impact for your customers is exactly the sort of thing that causes me to call it "unconscionable".

I agree that if it were just about the impact to themselves this would be "unconscionable", but I think that in practice the effect on users will also be very limited unless this vulnerability has already been known and exploited in the wild, or unless they take weeks more to patch OSX. Remember it requires a MITM attack to exploit it - not just a compromised host.

There is definitely a gamble they are taking, and definitely process improvements they should make (e.g. mandatory static analysis of shipping code). I just think think this is more about poor performance than poor morals.

[noting again that if this was discovered because it's in the wild, then it's all about performance]

The iOS patch was released on Friday, so it was publicly known then at the latest. It's not particularly hard to exploit, so it seems entirely reasonable to think that people were exploiting it soon after.

How many Macs connected to Starbucks WiFi this weekend and had their e-mail and banking credentials lifted? If my hat were darker, I'd have been out there doing it for fun if nothing else.

The fact that it requires a MITM means you're fairly safe on a home connection, but there are lots of people out there on public WiFi who would be pretty vulnerable to this.

I really don't understand what the "gamble" is. What's the upside to waiting?

I think your example of a Starbucks is a good one, and I'd be curious to know. The attacker has to go and physically sit in or near the Starbucks to do it, which is a strong limitation. They also have to have to have something worthwhile to do with what they steal. So that limits it to criminals with the knowledge to exploit this MITM attack who are willing to sit outside a Starbucks for long enough to harvest a worthwhile set of credentials.

I doubt that many credentials were stolen this way.

But some were, right?

Again, what's the upside to waiting? It looks like zero to me. Minor convenience for Apple at best. Placing their own convenience over the security of their customers is bad.

> Do you have examples of these far worse unpatched vulnerabilities? Not saying they aren't there, but I'd like to know what they are.

Search for what you can do for an Android phone that isn't on the very latest version of the OS, and then look up stats of how many Android phones are not, and will never be, on the latest version of the OS. That's unconscionable. To not backport fixes to devices, and to stop shipping updates to them mere months after their release in some cases.

> unconscionable makes it sound like it's more irresponsible than the industry standard - when in fact it is less.

> Yes, I agree that's a stupid call to make, but most smartphones have far worse unpatched vulnerabilities.

That's why Apple's response here is so interesting: it's much more common for smartphones to have serious and known unpatched vulnerabilities than it is for desktop and server operating systems to have them.

It's expected that a smartphone might go unpatched for a while, but the industry standard is that a desktop or server operating system should receive a patch for an extremely severe security vulnerability almost immediately.

Why Apple has chosen to ignore and invert that expectation is unclear. Was there an active, seriously damaging attack against iOS that they thought needed to be stopped right away, despite the cost to OS X users? Did they look at the install base and decide patching iOS first would have a larger impact? Is their development process for OS X not up to the task? Do they just care a lot more about their consumer electronics than their computers?

Then there's the question of why they published the details of the iOS patch before publishing a patch for OS X. Perhaps they rushed to patch iOS when they discovered the vulnerability without realizing that OS X is affected as well? The situation raises a lot of questions--while I'm not sure that what has happened is unconscionable yet (rather than merely extremely incompetent but well-intentioned), it's still an open question, and it's certainly possible.

My guess is that since on iOS every "browser" uses mobile Safari as a backend it was critical. On Mac, Chrome and Firefox do not have this issue, so it isn't as much of an issue. I'm not sure what percentage of Mac users use Safari.

Yes, nobody cares about their email or icloud account passwords being exposed.

Good thing they made sure all our AppleTVs are safe from MITMs, though.

Are you saying that since no one actually uses Safari it doesn't matter?

Or are you saying that this doesn't impact the underlying SSL library for all of OSX?

Or are you saying that no one (should) use Mac for actual work, so it doesn't matter?

Obviously, you're not saying these things, but what if you did. :)

The vulnerability didn't affect Chrome and Firefox, or my brew-upgraded curl. I wasn't too worried.

Apple lacks an emergency fix process. Also the fix will be released in 10.9.2, but it's held up in testing amgonst all the new added functionality.

I've seen quick reactions to security problems before. iOS didn't seem to have the same trouble. And if they lack an emergency fix process, that in and of itself is unconscionable.

> why it's taking so long to fix it on the Mac side

So, it turns out that they had a point release very far down the release engineering pipeline that included several new features (FaceTime Audio, FT call waiting, Mail bugfixes, et c).

A few days ago when the TLS bug became widely known they had two options: push a point release for _just that_ near-immediately, followed by the existing planned release (after redoing QA because the TLS changes would require the feature-add release be rechecked), or just integrate the TLS fix and restart QA on the planned release. They opted for making everyone wait a few days and just doing one release.

It was a trade-off, and both would have been okay decisions IMHO. There is a non-zero downside to releasing OS updates too frequently with an installed base as large as OSX.

"Unconscionable" it would have been if it had dragged on for a week or more. This was probably a heroic effort by the release engineering team(s) to get it out in in the timeframe that they did (after opting for just one release).

I completely disagree with your assessment of the two choices. Releasing an immediate security update independent of 10.9.2 was the only reasonable course of action. Holding off on a significant patch for days just because it doesn't suit your release schedule for unrelated stuff is ridiculous.

The only excuse for 10.9.2-only would have been if rolling a 10.9.1.1 would have actually taken longer to get out the door at all. But if that's the case, they have serious problems.

The best data point you'll see is ex-Apple people basically not defending the company, and any Apple departures in the next weeks/months.

If the bug affects Software Update, couldn't we use it to patch it ourselves? We could basically MITM our own machines to apply our own patch.

The above just made me think: This is a great datapoint in support of RMS and his rants against the dangers of proprietary software. Should we really be clamoring to some company for a fix, when we should just be able to patch it ourselves? (Should we choose to take the risk.) It's times like this when I feel like I don't quite own my own machine.

No need to MITM anything. You can just patch the binaries on disk. People have already done the legwork for it: http://www.sektioneins.de/en/blog/14-02-22-Apple-SSL-BUG.htm...

I tried it. Didn't work.

I enjoy that they're serving the patch over HTTP with no signatures or anything. So their patch may be just as useless or maybe make things even worse due to MITM

And they go out of their way to give you a version that's all scripted up for you so you can apply it without knowing what's going on!

You can always switch from Safari and Mail to Firefox and Thunderbird which do not suffer from this bug. As a bonus, they are also cross-platform, making it easier to switch to Linux or Windows later should the need or desire arise. As for Facetime, switching to Skype or similar will get you around the bug and permit you to chat, talk, and videochat with people that own technology from all sorts of companies... not just other Apple users (which is silly).

Right after reading this thread, I fired up Software Update, and OSX Update 10.9.2 is available for me.

Links to here http://support.apple.com/kb/HT6114 but nothing's on the page.

Edit: big update. 460M (I think), and took about 10 minutes on an Air.

There's now content at the support page you linked, but the security content of the patch still hasn't been published [1]. See the new thread about it here: https://news.ycombinator.com/item?id=7299287

[1] http://support.apple.com/kb/HT1222

For those who can get their way around a terminal, here's a temp fix:

http://nakedsecurity.sophos.com/unofficial-patch-for-the-app...

I applied it this morning. It works. The one on gotofail.com can't be signed so it doesn't work.

This patch still doesn't solve the real problem but at least it doesn't fail silently.

If my debian system were to break, and noone was around to fix it... I could fix it myself. Free software ftw.

The security flaw is inside a library that has been released under a bsd style license (otherwise, the "goto fail;" hilarity would never have ensued). You're free to download the source of the 10.9 library, patch it, compile it and replace the vulnerable binary with the one you fixed.

Apparently someone tried it and the publicly-available source is incomplete and doesn't build.

I think what's interesting is that this code is open source, in code if not development model, but it failed the law that "given enough eyeballs, all bugs are shallow." Until there was an inkling of trouble, at least, and then it was quite shallow indeed. So I wonder if white hats will now look at opensource.apple.com more routinely, because I'm sure black hats are there already.

One trouble with that is that the "source dump" style of open source that Apple engages in doesn't really attract eyeballs very well. Sure, you can go read the source, but it's hard to do much with it. It's hard to tinker with it, since Apple doesn't provide any good facilities for installing the stuff into the system. There's no place to send patches. You can send in bug reports if you find anything, but why would you bother when it's so hard to contribute code?

If you're interested in security and hacking on security code, OpenSSL would be a much better choice just because you can potentially become part of it, not just an observer.

If the system were to break, and nobody was around, you'd only know there was a problem when you were so compromised that your machine failed.

OSX machines are now iOS development workstations; nothing more.

What do you mean?

Apple can't be trusted to do point releases for major security bugs in a timely fashion.

OSX development can only be done on OSX.

Because Apple security procedures are now known to be so horrible, the reasonable thing is to only use Apple hardware when you absolutely must -- iOS dev.

I say this as someone who currently has only Macs except for servers; I'll probably not buy another one, and switch back to Linux. I might Linuxify the Macs I currently have, except for when I need to do iOS stuff.

Correct me if I'm wrong, but you're almost completely abandoning an OS just because of 1 security problem?

I don't actually care about the original bug much. It happens.

That Apple's internal code review/static analysis/etc. doesn't exist is a bigger problem, but still not a showstopper.

That Apple's incident response and prioritization is horrible is the reason. Look what they did with the dev center over the summer. Various past bugs.

You say this as someone who does not understand the issue at all and has a very naive take on it. Alas, voice of tptacek was not heard on this issue for some reason :(

Fortunately we have your extensive experience in operating system security patching and policies about when to push a hotfix vs. a large update to a widely-deployed userbase to enlighten us!

Am I the only one who despairs over redundant hashtags being appended to everything?

Yes #iforonewelcomeourmemeoverlords

... am I the only one that thinks "pound" would be a better phrase than "hashtag" when saying these out loud ?

Cuts syllable usage in half ... has worked for decades with irc channel names ...

'#' is commonly called either "hash" or "pound", and "tag" as a suffix just describes this tagging use in context. For an apples-to-apples comparison you'd be saying "poundtag", but good luck getting that to catch on. :)

Octothorpe FTW

Hmm over here on this side of the pond a pound symbol means £ not octothorpe / hash / sharp.

Tossed an email to my AppleCare contact expressing my frustration... You should too if you have one!

I love how people keep making excuses for why their favorite cult leader just fed them cyanide.

Looks like this should be updated - 10.9.2 was just released.

http://www.macrumors.com/2014/02/25/osx-update-ssl-facetime-...

I blame the security community on this one, for not releasing an apocalyptic weaponized exploit for this vulnerability over the weekend, instead of stuff like agl's checker.

If end users were on fire, Apple might be more motivated to push a fix.

...and it's fixed. http://www.macrumors.com/2014/02/25/osx-update-ssl-facetime-...

I can't afford to switch environments at the moment. I don't however use any apple applications such as calendar/reminders/safari. Does this mean I have a modicum of relative safety?

https://www.imperialviolet.org:1266/ produces an error for me. And from what I gather that means I am at least safe using google chrome on OSX.

Regardless of the fact that you, and I, realize that two separate teams are working on these things, it looks really bad (well, at least to me) to have your flagship OS vulnerable to an amazingly easy to exploit security hole for multiple days, widely and loudly publicized ...

And nothing comes out. Oh, except for iBeacon, a specification for pushing ads on you based on your location.

Tim Cook had better make a public statement and make it soon.

Think antennagate.

It's one thing if your maps application is wrong ... but it's quite another if suddenly people feel like using your product puts their banking information at risk.

I really hate these long-url one-word-of-content sites. Doineedajacket.com was clever, but the swarm of copies are unaoriginal and annoying.

Looking at the details of the bug, I'm surprised it wasn't flagged with a warning. Why don't we warn on unconditional gotos?

Just curious, what Linux distro is everyone switching to? At this point I am seriously considering it, because this is pathetic. (I suspect I'll remain with Apple hardware for the foreseeable future, because they still have the best laptops IMO, but running another OS is not out of the question.)

I like elementaryOS, but it really doesn't feel as polished as OS X. Things like their choice of font don't help IMO.

If you're seriously considering switching to Linux, then be aware that nothing you find is going to feel as polished as OS X. Linux developers tend to be more focused on security and under-the-hood improvements while Apple focuses on user experience, plus Apple is a business that can easily afford to hire as many developers as they need while most Linux distros are community-driven.

That said, the Linux user experience has improved dramatically over the past several years, and my recommendation would be openSUSE (what I run), or Ubuntu if you're completely new to Linux.

Yeah, this is what I was thinking. elementaryOS seems to get the closest but still isn't ideal. I have to think about it. Thanks.

I use Linux on servers, but I've always found the options lacking in some way for desktop.

Debian jessie.

I'm a fan of archlinux running the cinnamon desktop environment. Archlinux might be a bit intense depending on your command line fu; you might want to try Linux Mint instead.

Hardware wise, Lenovo also makes decent hardware. The Yoga 2 is a pretty solid machine comparable to most of Apple's offerings.

Personally, I run Debian Wheezy in its default desktop install, GNOME3 and all. It works very well in my opinion, but is neither as shiny or "solid-feeling" as OSX.

opensourceapple.com ? If it's an open source part in apple code, then couldn't I fix the issue on my own machine? (by removing the second goto fail; and recompiling)

In theory yes (especially since it's a framework, as it's dynamically linked against), but download [0] and see the README: you'll be missing some proprietary algorithms so some things depending on them are bound to fail.

[0]: http://opensource.apple.com/tarballs/Security/Security-55471...

IOS 7.0.6 This security update provides a fix for SSL connection verification.

Just notified on my iPad.

That was 5 days ago. OSX 10.9.1 is still vulnerable.

Since all of the computers in question are Intel-based, I suspect it would be possible for people to use bootcamp to run Linux or Windows. Are people switching over?

EDIT: apparently I have to spell it out: if people are bothered by the situation, they will switch to a different OS. And since we are talking about OSX on computers with intel chips, that is an option.

Yes. Everybody has switched.

I think simply not using Safari will mitigate most of the impact for most users on OSX.

Reply on news.ycombinator.com