Now Resolved: FB iOS SDK Outage Causing Disruption to 3rd Party iOS Apps | Hacker News Reader

Now Resolved: FB iOS SDK Outage Causing Disruption to 3rd Party iOS Apps

developers.facebook.com

26 pointsdfabulich21 comments

I clicked, pleasantly surprised and expecting a post-mortem. Instead it's a corporate PR admission of guilt and non-apology. If your audience is developers who implement the SDK, a more detailed explanation is the only respectful option.

Moreover, the actionable advice to upgrade the SDK is a complete non-technical non-sequter seemingly put as a distraction:

- They admit it was a server side change that caused the crashes. SDK version doesn't change that.

- No breakdown of which versions were affected (or was it everything?).

- Is a newer release safer? Were changes made that we would get by upgrading?

Did anyone by chance MITM the server payload to understand the bad code/bug?

the latest version (7.x) was not affected by this. we had a build that was working, but it’s impossible to release fast enough to matter.

I fully agree with you that they should have put out a post-mortem, that's what I was expecting as well, when I clicked the link. I disagree on it being a non-apology however, since the sentence below, quoted from the article, seems like an apology to me.

> This is the second similar incident this year. We want to apologize for the inconvenience these events have had on our developers and their users.

It’s not much of an apology because the final paragraphs try to push any future responsibility onto the SDK consumers by saying they should make sure they are updating as fast as FB would like them to.

It’s an odd addendum to put in and appears ominous to me.

If they want to push that responsibility they should do it officially rather than as a suggestion. The company I work at has a concept called a "build horizon." The rule is that clients have to be updated within the horizon. Internal services do not have to consider internal clients older than the horizon, and may break them arbitrarily. In effect this means that even the least with it internal clients rarely let their internal releases languish more than about half the length of the horizon, because of the associated risks of missing the cutoff.

There are many similar successful concepts that cap the maintenance window that services must tolerate, while at the same time giving developers enough forewarning to understand what is expected of them.

Wanting to do something isn't the same as doing it.

Are you saying that because the author said "I want to apologize" instead of "I apologize"? I don't remember how this phenomenon is called but basically in English and other Indo-European languages there are expressions that don't really modify the meaning of a sentence, another common example is "you know, I think I'm going to have a sandwich" vs. "I'm going to have a sandwich" - the two sentences have the same meaning, the speaker is not really asking you whether or not you know that they want a sandwich. This might be confusing at first, depending on your linguistic background.

Time to get rid of this huge and buggy SDK. Just to offer FB login to apps it‘s kind of ridiculous this is required in the first place.

Not an app developer, so genuinely curious: would offering only Google and Apple SSO (alongside email and password) be enough coverage for most users to reduce signup friction? Apps are rolling Apple auth out now as the deadline for implementation has passed.

My understanding is that to use Facebook as an auth mechanism, you must use their SDK (and personal data vacuuming along with it), so dropping Facebook auth would allow apps to drop the SDK, which would be a win for both privacy advocates and users who enjoy their apps working even when Facebook breaks something.

> My understanding is that to use Facebook as an auth mechanism, you must use their SDK

Does anyone know more about this?

As far as I remember Facebook does support being and openid connect authentication provider so it should be theoretically possible without their SDK. But it might be that they put some practical restrictions in place to force usage of their API??

I believe the T's & C's of the FB developer platform specifies that you can only use the Android/iOS SDKs for auth. You can probably get around it using openid, but you risk your developer keys being revoked.

Lots of login providers are out now. Okta, Auth0.

The relevance of upgrading is that the latest version didn't crash. While they clearly don't test server changes against every version of the SDK, they hopefully at least test it with the latest.

The crash was just that a dictionary decoded from json sent by the server contained null in a place where the code expected a non-null value. The actual payload difference would not be very interesting.

> If your audience is developers who implement the SDK

I believe the audience is marketing and management.

> This is the second similar incident this year. [...] While we can’t guarantee a bug-free experience, we have learned from these events and continue to improve the stability of our SDK.

What did they learn, exactly? There wasn't a postmortem on the prior incident, and it would seem that no postmortem is planned for this incident, either.

A postmortem has been requested; whether we'll see it... who knows.

https://github.com/facebook/facebook-ios-sdk/issues/1385#iss...

If there’s no postmortem I think it’s pretty safe to assume the mistake(s) were embarrassingly simple such as a disturbingly poor CI process for a multi billion dollar company.

There is no shame for a bad config push... it happens... but there’s a huge amount of shame in having an ego so big you cannot admit it.

When I worked at FB, I once brought down provisioning of bare metal fleet-wide for five hours because I forgot a semicolon in the .bashrc of the boot ramdisk... the linter didn't catch it because it only went off of file extensions, and ".bashrc" didn't end in ".sh". Even huge behemoths aren't immune from CI and linter issues.

Agreed on your second point as well.

They invented the "move fast and break things" attitude.

That's fine for a social network, but by forcing the SDK to be directly integrated into third party apps, they're forcing their attitude onto developers that have a "move slowly and don't break things" attitude.

I'm shocked that they have this rule, that developers play along, and that there's apparently nothing that can be done about this.

If there was some easy way to find out which iPhone apps that I'm using have the FaceBook SDK installed, I would mercilessly uninstall all of them.

But as a consumer, I'm completely blind to the privacy-violating malware that is being forced upon me by faceless corporations.

Yay.

What I've learned is that I'm never putting a line of Facebook code into anything I develop.

I don't think anyone wants to put the FBSDK into their apps. You kind of have to to optimise your cac.

Reply on news.ycombinator.com