App suddenly crashing on startup due to FBSDKRestrictiveDataFilterManager.m
github.com
github.com
WhatsApp, a Facebook owned app, will not give you read access to your own chat database - it’s encrypted. But they have an arrangement with Google where it’s will back up unencrypted to your Google account (though, Google won’t let you retrieve it ... only the WhatsApp app can)
Everything about this duopoly, their cooperation and lack of competition stinks to high heaven.
On android? On iOS you can get a tarball of all your chats.
As much as I'm happy to pile on Google, Facebook and WhatsApp, this appears to be untrue.
But you can export all chats one by one.
The statement that "Facebook gives an unencrypted copy of your chat database to Google but will not give it to you" is true; I'm referring to the SQLite database. You can back it up locally, but it's encrypted and Facebook will not willingly give you the key under any circumstance (though it is possible to trick them by impersonating the app). However, the Google Drive backup is unencrypted. But Google won't let you download it directly either (although, again, you can trick them by impersonating the app).
if that's the case then facebook can't "give you" the key, because they don't have it, it's supposed to be on your phone and only there..
Is there something i'm missing here ?
They make a copies of that plaintext database for backup purposes:
One on google drive (not mandatory, but it IIRC by default), which is still plaintext but not accessible to yourself.
One on your local SD card (if you ask for it), which is encrypted with a key that's on your phone but was sent down from FB servers. If you switch your phone, and try to restore that database, it will contact FB servers to retrieve the key.
> if that's the case then facebook can't "give you" the key, because they don't have it, it's supposed to be on your phone and only there..
Regardless of the backup encryption, e2e doesn't mean that they don't have the key; it just means that it leaves one phone encrypted and enters the other encrypted without being decrypted along the way -- unlike, e.g. regular email, which is usually encrypted in transit by TLS, but gets decrypted and re-encrypted at every stop along the way.
Recently I was trying to switch my wife away from the free Keeper app to the one I use and backup to our own storage, they make it impossible to get your passwords without paying them or rooting your phone. I hand copied them one by one from her old to new phone. Honestly, screw any company that does this.
When you make a local backup, it is encrypted, and Facebook doesn't think you deserve a copy of the key or an unencrypted copy.
Don't ask me what's leading WhatsApp's legal department to believe that this feature doesn't comply with German privacy laws. It's quite puzzling. (I would've rather expected a feature like this to be mandated by EU laws.)
As of now, there's no non-hacky way of exporting chats for German users. (Hacky ways include: dubious third-party software, copying each message individually, taking screenshots.)
In ecology, it's rare to species compete to extinction or exclusion from a niche. It's much more common to observe segmentation. It's a lower energy state for birds to mate at different times, eat leaves from different parts of the same tree, etc., dividing the resources of their niche among themselves, than to force each other out of the niche completely.
The situation is analogous. I'm sure they'd destroy each other if they could, but it's obviously not happening right now. It's much easier to say, you can have your current users, we'll have ours, and we'll focus our energy on capturing emerging markets. Which is why you see them racing to provide free internet to developing countries and such.
Wiping the WhatsApp chat history on the old phone before giving it to a new owner is also much more important. It often contains highly embarrassing or even incriminating details.
The net effect is that if you're switching you have to redo 2fa for all accounts where you haven't separately backed up your TOTP codes.
You can store data on Google Drive that isn't accessible from Google Drive and only that app can read or write it.
But why would WhatsApp store it unencrypted on Google Drive, and encrypted (without giving you any access to the key) on your own SD card or local backup? It makes no sense.
Except ... Google and Facebook have agreed that WhatsApp stores the database unencrypted, and in return Google doesn't count it against the user's quota. Everybody wins! Facebook get to have free backup for users, Google gets to see all user messages and metadata. And the user gets rid of this pesky privacy thing!
[0] https://faq.whatsapp.com/android/chats/about-google-drive-ba... - "WhatsApp backups no longer count against your Google Drive storage quota." and "Media and messages you back up aren't protected by WhatsApp end-to-end encryption while in Google Drive."
But, you know -- they gave the same kind of info (and much, much more) to firms like Cambridge Analytica for free for years. So perhaps they didn't even get more in return.
Facebook's purchase of WhatsApp was OKd by EU regulators subject to some constraints, e.g. that they don't use WhatsApp data to enhance FB graph and/or gain other unfair advantages. FB has since admitted to using WhatsApp metadata in violation of that constraint (and didn't even get a slap on the wrist) - and perhaps giving the data to Google is also part of playing dumb. I don't know, but I'm sure it has everything to do with benefit for FB and GOOG, and almost surely at the expense of user privacy.
The design is pretty clearly to support the bog standard backup use case of "something happened to my device, I got a new one, I want my old data".
It would be nice if they added an option to also encrypt backups, although I think it would be non trivial to design where the key material would live. (phone enclave? Multiple devices? adding/removing devices? etc) Not to mention just having the option might cause more harm than good if people turn it on and regret it.
I think their design philosophy here is likely "let's err on the side of smooth UX and let the 'end-to-end encryption is srs bzns' loonies go to signal"
They already support encrypted backups. In fact, they don't support unencrypted locally.
> The design is pretty clearly to support the bog standard backup use case of "something happened to my device, I got a new one, I want my old data".
They had, in fact, perfected the local encrypted seamless "something went wrong" workflow - they keys are delivered from the server (stored or generated, I don't know) once you've proved you can receive texts on that phone number. Without that, the local backup is useless.
And IIRC, they ALSO did the same with Google Drive. But then, one day, it stopped counting against your quota and the encryption disappeared.
> I think their design philosophy here is likely "let's err on the side of smooth UX
The UI was as smooth as it can be. It still is, for local backup.
I try not to attribute to malice that which can be explained by stupidity, deadlines or missing budgets, but in this case, there was a decision to remove encryption, that required negotiating with a rival. That decision had costs, and the process had costs. There is a benefit associated or this decision would not have been taken. I don't know exactly what it is.
On one hand - even after end to end encryption, we continue to see WhatsApp messages quoted in prosecution including in SEC actions and prosecutors are clearly getting access to messages even pre-arrest. The ability of prosecutors to compell unlocking after arrest in US is questionable. So how are they getting access to the messages if there's truly end to end encryption? Also there are any number of Governments in the important non-US markets that would simply ban WhatsApp if they didn't have access against a warrant (eg. See BlackBerry cases from some years ago)
On the other hand, no security researcher seems to have found or at least reported a back door in the WhatsApp apps that supports MITM undetectable to physical key verification which is the obvious weakness in an end to end encryption system based on centralised key repository system.
On the gripping hand, all high profile cases trying to compell tech companies to provide evidence have been against Apple. I've not heard of one against WhatsApp.
Ergo - the most like scenario is that this feature is deliberately built keeping messages un-encrypted so that when law enforcement shows up with a warrant or a muzzled warrant, they can just hand over the archive without weakening their app. People are convinced to opt in because it's the only way to change devices while keeping your messages. To give people comfort that unencrypted messages ate safe, it is sold as as "it's in your own Google drive"
The Google Drive backup is most likely still encrypted, just not "end to end" because the key needs to be stored on a Facebook server.
That makes a lot more sense to me.
Convert all active whatsapp contacts to signal? 80%? 50%?
Same for Facebook messenger!
It needs to be uncomfortable enough - if they want to communicate with us and are minimally tech savvy ("click link to install app") they have to switch to signal.
That's it.
I'm starting to send out messages over my morning coffee.
Let's do this together:
https://support.signal.org/hc/en-us/articles/360007060592-In...
The result is uncomfortable, since Signal never knows the other side removed it, so messages land in thin air. Several times did I have awkward moments when someone was angry "I thought you'd let me know if it was cancelled" - "I did - I sent you a text about that two days ago! On Signal!"
And after it started insisting on creating a PIN each time I start Signal, I finally deleted it from my phone.
You can forget about security if you can't get a seamless user experience.
Plus, it's just a terrible UX for a SMS client. I really wanted to like it, but it's just too obnoxious to use. I've certainly not recommended it, I've had better luck switching relatives to Linux mint (seriously, I am not exaggerating). Creating a group MMS is annoying, and there's a nag screen if you try to call someone from a message within the app. I understand that most of these problems are due to the secure nature of the app, but trying to use it as your full-time SMS client is just too much of a pain. I'll likely still keep it installed, as the 1 thing it does do is make it really easy to transfer files/send messages between devices (send messages to yourself, then download them using the client's for other OS's).
Personally, I don't feel the UX is "terrible", I can use it to do most of the things that I could do in WhatsApp. My friends and I also tested out the video call feature and found the quality to be the same, if not better.
That has never happened to me.
At this point, none of us even use phones, my family pretty much use Zoom and at work everything is Google Meet, my phone function is basically a dead function at this point.
My email represents me, and I fully own it with minimal upkeep costs.
You can make the argument that I don't truly “owns” the domain name off of which my e-mail address hangs, either, but it's a much better situation than phone number ownership. There, at least, the constraints of ownership (rental) are clearly defined, as is my ability to move it between registrars.
Yes I think they suck, but do they suck more or less than en email? Not clear.
I'm a core contributor at Status, and the team I'm on is currently developing a desktop client[1] to complement the mobile client. We have alpha builds for Linux and macOS, and soon Windows.
Status uses the Waku[2] protocol.
One thing: I don't think it's unusual for a user to have little/no interest in the wallet or dapp browser. In that case they simply use it as a chat app that's focused on privacy and security while ignoring the other features.
The uptake? Like ~30% of the people I'd like to contact bother with it. The rest, it's even worse - we get stuck with SMS. And a lot of conversations I just miss, period, because they're groups of people in one of those other ecosystems (and missing those is a price I'm personally OK with paying, but not everybody will be).
The situation is just really bad. Signal is a good baby step forward, but it's not really the answer for most people, because they're sticky to wherever their network graph of contacts already is, and they're not going to switch just because the one weird nerd in their life isn't available there.
are you OK with THAT? You support children harassment by cops?
Throw some biased media page calling big companies bad. Not a hard thing to do today or write an opinion piece on popular news journal site and reference it. Medium works too.
We also need to cancel people for supporting these platforms.
They can start by removing the atrocious phone number verification first.
Being forced to give them one of my 'most unique identifiers' (my phone number) was an absolute 'no'.
There are some good-looking and user-friendly clients
1. There are tons of replies in this thread. People have very strong feelings about signal.
2. Some people feel VERY STRONGLY that the pin numbers are stupid, and it has a terrible UX.
3. Some people feel VERY STRONGLY that the phone numbers are the simplest, most straightforward thing to use, and they would hate Signal if they didn't support it.
4. Some people feel VERY STRONGLY that they don't want their phone number used as an identifier, and it's a failure of Signal that they use them.
And it seems like many of these replies assume that their own position is the only possible one to take (or at least the only intelligent one!).
Imagine working on Signal and trying to create a product that pleases all these people, as well as even less technically minded folks, while having none of the usual analytics at your disposal to measure actual user interaction!
My hat's off to Signal. I admire their work but I don't envy their position.
Or do you demand that any file under your control that contains chat should be accessible, rather than just one of them?
You can't download it without impersonating the WhstsApp app.
Do you know if the icloud backup is encrypted or not?
1. Shady addictive game requires full access to drive.
2. Shady game downloads all your chats with all contacts
3. FB is on fire since they allowed access to people's chats with others, even those who didn't download the game.
4. People get mad on FB/Google for not protecting their data.
Claiming that this is the reason is sort of like rearranging the chairs on the deck of the titanic when you can see the iceberg.
Much more likely it is to stop import/export to a competing app. The fact they will let you keep a local encrypted copy, but not an unencrypted one under any circumstances, is telling.
If we are to take the position that all users are tech literate and should be fully in charge of how their data is used, then even Cambridge analytica wasn't bad. People explicitly gave it access to data that was accessible to them. However, the uproar proves otherwise. People don't like it when they click yes yes and some random app has access to their social media data or messages. This is why Gmail also started requiring independent audit for all gmail apps.
SMS had the same problems forever, but isn't owned by a company, thus you can't blame xyz company if random apps access your SMS. No company got ever investigated by multiple nation states over that.
You still haven't addressed my main question though, how will a random app scraping your WhatsApp be any less of a scandal than Cambridge Analytica?
Haven’t seen anything like this before.
some serious cognitive dissonace is going on - either that or it's the effects of the news-cycle making us blind to it. Maybe we should think twice before applauding FAANG engineers in future posts because Technology isn't neutral.
note: a company can't be evil. it's just a group of people who can be evil and who hide behind the facade of the "legal person" statute. I'm happy to celebrate individual breackthroughs but let's not forget that without (metaphorically) nailing the culprits to the cross who are responsible for ICE contracts or privacy invasive tech there isn't even a point in complaining about all the horrid stuff these people (working for these companies) do.
That said, with FB/Google these days the bad weighs more than the good.
just because we made the mistake to let them become this big without giving them criticism doesn't justify why we should continue treat them with gloves. quite the opposite. they shouldn't feel comfortable at all in their position - they should feel like they're being held accountable (and act accordingly whether with rebelling internally or by leaving depending on the size of their spine)
yes - and I've done incredibly well during it. I also know change doesn't come for free. the sad thing is the bigger the companies the smaller the spines of the individuals working there.
We all put thoughts into our own well being. When sufficiently comfortable or successful at that we start caring for people we interact with on a daily basis. When that too is a reasonable success we start doing stuff for people we interact with less frequently. At some point you can chose to go look for people who need our help, do something for the town, the city, the country, humanity etc There is tremendous joy to be had in this process and it builds character. I admire people like that.
People who don't do it are missing out. They in stead keep telling themselves they can't do anything for others knowing they are lying. The endless rationalization of the irrational takes a huge amount of time until they graduate themselves in the DIY course in psychopathy. In their ego trip they end up feeling sorry for themselves. In their self inflicted blindness to what misfortune looks like the self pity knows no boundaries. Also: no one loves me! I wonder why that is? They keep trying to spend money to buy happiness but all they get is cheap thrills. Some end up in a pathetic cycle where all they can do is expand their wealth, ignoring even their partner and their kids. They don't have time for that, they have important other things to do. I pity people like that. If I can think of some way to help them with their condition I will do the responsible thing.
It's like Kierkegaard's example of the mob who collectively intimidates but every member argues they have contributed virtually nothing, and so any sort of responsibility is simply dissolved.
Anyone who is responsible needs to be aware of the holistic system that they're part of and that they are enabling rather than just their own little world. That's no excuse.
Consider this menu of possible ways to be associated with a human rights violation:
- Do it yourself.
- Be a part of the chain of command that ordered it.
- Be commanded by the chain of command that ordered it.
- Be part of the same organization as the one that contains the chain of command that ordered it.
- Be a member of the society that contains the organization that contains the chain of command that ordered someone to do it.
- Be a member of another society, that could have tried to stop it, but didn't.
Where do we switch from participant to accomplice to bystander to "completely uninvolved?"
First they came for the socialists, and I did not speak out because I was not a socialist.
edit: In case it isn't clear: You just chose sides right there in your comment.
But they also produced some really cool tech. Unix being the obvious one to bring up on this site.
That era is romanticized to hell (including by me), and Ritchie, Thompson, and the like are folk heroes. Despite being engineers at the subsidiary of a giant monopolistic entity.
Ultimately, for people who think the tech is cool, it doesn't matter where it comes from. In 50 years people will be romanticizing the era of the FAANG companies (more than they already do) the same way we romanticize bell labs.
At least this time you can disable the SDK (at least one app developer has a kill switch for the SDK) unlike the last issue that happened during code initialisation you couldn’t skip.
"A distributed system is one in which the failure of a computer you didn't even know existed can render your own computer unusable." -- Leslie Lamport, 1987 (https://lamport.azurewebsites.net/pubs/distributed-system.tx...)
There's no such thing as QA at Facebook. Really. Developers are supposedly responsible for all testing of their own code, but all of the pressure is to "move fast" so that goes about as well as you'd expect.
The first two times it took months to fix. It’s happening again currently, too.
Clear there are GAPING holes in their test suites.
Make a lot of money selling lifetime subscriptions, then have nowhere to go for cash next year: look at all that money! I'm very successful now!
If all you're measuring and optimizing for is "valuation" or "revenue" you're not automatically going to get a good product or happy customers...
If you have a team run by idiots, then you won't have QA.
If you have a team run by adults, then there is plenty of QA. I mean its total shite that this wasn't caught. I'd expect much better, especially as this is the second time in n months. But then perhaps working for a company run by a dick who can't listen is getting to people.
Engineer in test. QA engineering lead. (https://www.facebook.com/careers/jobs/?q=QA) You must have leet scuba skills with searching skills like that.
AR/VR, Oculus, portal and workplace all have dedicated QA engineering teams. I'm sure there are more.
> Given your characterization of MZ
if it was only zuck. Zuck has two faults: he's not able to articulate any single thought inside of 15 minutes and he consistently throws out integrity products that would have stopped bad PR.
Its not just me, you should look at the micropulse. confidence in leadership has taken a good battering recently.
The whole juneteenth buisness, we had an entire _day_ of people telling us the problem. Senior management then sat next to them and said in reverent tones: "thank you so much [[dabs a tear]] for telling us your lived experience" posts a selfie and then does fuck all else.
Or, having revved up the company to generate hundreds of ideas to tackle abuse on FB platform, do shit all with them, not even acknowledge they exist.
Nothing to do with scuba, and if you did any operational work you'd know that it's in the category of tools that are often flaky due to lack of QA. It's also worth noting that the link you provide only shows 41 listings, and most of those merely mention "familiarity with QA" in some boilerplate for what's obviously a developer position. Leet reading skills you have there. Percentage-wise, there are fewer true QA positions there than we have E-10s. For a company that hires 10K a year, that's indistinguishable from zero.
> AR/VR, Oculus, portal and workplace
In other words some small percentage of the company, none of it near me. All of the systems that the profitable parts of Facebook rely on - to provision systems, monitor systems, store data, move data around, analyze data, etc. - rely entirely on developer testing. So here's an amended statement to make you happy.
"99.9% of Facebook has no QA, and the rest has really bad QA."
just over 10% of job openings have something to do with testing or QA. Of course QA is similar to software engineering, the whole point is to automate testing.
> [AR/VR, Oculus, portal and workplace..][..is] some small percentage of the company
yeah, go look at the headcount.
> none of it near me.
Now we are talking.
> rely entirely on developer testing
again, that wasn't the distinction I was making. It doesn't matter who's testing, so long as it gets done properly. Considering PE is a blended role. Its not a hard push to think that testing might be part of the role of a modern developer. Mind you, so is writing documentation, but thats not a priority either.
No, I don’t think there’s any “to be fair” in that, unless those SDKs have been officially EOL’d. Even then it shouldn't take down the whole app, Facebook should just break.
That this only happens with some SDKs certainly makes the problem less severe than last time, but only relatively.
Workaround for developers: Go to Facebook Developer page > Analytics and disable automatic Events.[0] Apparently this doesn't solve the issue for everyone.
Issue in Facebook's tracker: https://developers.facebook.com/status/issues/17391881029111...
[0] https://github.com/facebook/facebook-ios-sdk/issues/1427#iss...
facebook.com
fbcdn.com
fbcdn.net
fbsbx.com
fb.com
instagram.com
https://github.com/confirmedcode/lockdown-ios/blob/master/Lo...
https://github.com/confirmedcode/lockdown-ios/blob/master/Lo...
https://github.com/confirmedcode/lockdown-ios/blob/master/Lo...
https://github.com/confirmedcode/lockdown-ios/blob/master/Lo...
I couldn’t help but notice neither app worked even after a reboot, yet google maps did. I was thinking there was a carplay bug and then thought: “oh boy here we go, what are the odds, let’s check hackernews” and bam, of course it’s the Facebook sdk again
I’m thankful that googles iOS-chrome and google maps doesn’t use the Facebook sdk for something...
I want a central document with clear steps on how to mitigate. Currently the best option for documentation seems to be wading through comment threads, which is very frustrating.
Or if you must, use their oauth flow and API but _don't_ include code of theirs you don't control directly in your binary. It's just asking for trouble.
We had nearly 100,000 users logged into our app via Facebook login so we were basically strongarmed into complying. We somehow pulled it off before the deadline for both Android and iOS but felt really dirty afterwards.
>It turns out that by just including the SDK with your app, Facebook runs hidden code on launch. (FBSDKApplicationDelegate.m)
[0] https://twitter.com/sandofsky/status/1258288056399847425
There are some intricacies on iOS surrounding null values and serialization, and as a developer it is important to understand that you may encounter NSNull. As a standard practice in my company, we type check every remote value before calling anything on it. Seems like that would have prevented this issue.
Source: have had that issue before
Also maybe it’s time for them to start rewriting things in Swift, where this would be much less likely to happen
It makes me wonder what all those engineers at FB are actually doing... ? Every time I tried to integrate or look into the FB SDK for simple functionality it was a total clusterfuck.
A bit of snark but also maybe stop trying to use a data collection engine to leverage anything for yourself?
If Apple wanted to make money , it should make it harder for advertising supported apps to make money and force apps to either not exist or get people to pay for them.
I would be okay with that.
What has this industry become? How are we so goddamn inept at writing software? This is an industry where we can automate testing - not many other industries have this capability - and we still don't have a simple test regime of "check if the app opens" for any of those apps. Somehow, at the most valuable software companies of the world, nobody has set up a system that makes sure that you can't introduce a change that renders these apps unusable?
WHAT THE FUCK.
I have always been afraid of software engineering becoming a trade similar to medicine or law or (regular) engineering where significant barriers will be put up before you can enter the industry, but this sort of shit makes me feel like it could be for the better. I will have to go and get a degree, but that's the price I have to pay I guess.
A couple of days later I caught up with the people allegedly responsible for this stuff in the bigger review meeting. "Shouldn't you open a SEV because you are shipping stuff that can't possibly work and aren't catching it pre-release?" The response was a lot of hand-waving and stupid manager-speak that boiled down to "no, go away".
I basically wrote them off at that point. If they could act that way and get away with it by way of implicit acceptance from upper management, that's just the way it was going to be.
I see the quality is as high as always.
:(
Facebook's own apps did not break since they are on the latest version of FBSDK.
What Facebook needs to do is to test whether their changes don't break other people's apps.
If hacker news goes down while you are writing this comment, it is your responsibility to ensure you can still post your comment despite the service breakage.
Many of the apps crashing are using Facebook for a portion of their app, such as one login option.
Letting that take down the whole app is insufficiently defensive coding, whether in the app, sdk, or glue between them.
Absolutely, but the problem is that in order for you to use these features, you're supposed to use the official Facebook SDK. The additional problem, is that Facebook has no interest in defensive coding, because if you're not interacting with Facebook in an app, there's no benefit to Facebook for you using that app at all.
(Am not a mobile dev, but it wasn't exactly hard to find that, either).
Other than remove the SDK, of course.
Fool me once, and all that.
Bingo! App developer voluntarily pulled in a dependency which they didn't fully understand. When you bring in a dependency, it's not still someone else's code/problem, it becomes your problem.
2. Client-side SDKs can be designed in a way that even if the server-side changes (which it shouldn't), it does not cause the whole application to crash. This can of course be hard depending at which level the SDK operates and how integrated it is into the app.
3. Are there no contracts between Google/Spotify/etc and Facebook? If I were FB and there were contracts on the line, I would make sure that there is no way something like this can slip through (automated testing).
I'd argue this is at the very core of why you test. I mean this problem is as simple as (n+1).m vs n.(m+1) in terms of versioning vs revisioning.
If that is not being tested then I'd say Facebook isn't actually testing at all - at least not in a practically relevant sense ...
... despite the shit-load of YouTube videos from FB devs lecturing about CI/CD and what not.
2. Throw exceptions
Don't tell me "No amount of testing" just because you aren't responsible for your own dependencies.
Removed the SDK myself from a top 100 App Store app.
This will be a fart in the wind and lessons will be learned.
The industry seems to have very low standards. Let's let these mistakes happen over and over and let's just patch them as we go. No long-term thinking.
Someone should collect all these horrible incidents and start a university course on just plainly what NOT to do.
Don't sit in an ivory tower and throw stones in glass houses. Can you guarantee that your own SDLC house is in order? Because it might be, right up until a sneaky edge case nips you in the butt, and then suddenly it isn't.
They're using static initialization and it bit them in the ass again. They should stop using it because it is prone to things like this which result in apps not launching at all. I'd like to see them commit to stop using this method to initialize themselves, and take responsibility for the crashes. Then, IMO, we'd be good.
In a scenario like this, I'd hope they'd want to make this bug's recurrence impossible.
Here is our Android config in case anyone's interested:
<meta-data android:name="com.facebook.sdk.AutoLogAppEventsEnabled" android:value="false"/>
<meta-data android:name="com.facebook.sdk.AdvertiserIDCollectionEnabled"
android:value="false"/>
<meta-data android:name="com.facebook.sdk.AutoInitEnabled"
android:value="false"/>Why should we just sweep this under the rug and call it a day? There is something very important that can be learned from this. Apparently, FB themselves have not learned, because this is not the first time this has happened. Why was this allowed to happen more than once? Did Apple, Google, Spotify, Facebook etc not take any actions?
"Shit happens" is one way to look at it. Another way is to question whether we as software engineers are really doing our job properly.
> FB themselves have not learned, because this is not the first time this has happened.
They have tens of thousands of engineers likely split into hundreds of different teams/microservices focussing on different parts of their software stack. A ton of them are new to it and nobody knows every part of the stack so shit can happen.
What is the biggest engineering organization size you have worked for and what was your uptime?
The linked issue makes it very clear that the "tiny outage" affects literally every app that has the Facebook SDK merely linked, for a frankly dumb reason (static initialization) that they should have learned from the last time it happened.
It's not a SaaS. "Uptime" for a linked library is stupid.
Like they work in many other companies.
In any case, this is not about being bright but about being diligent.
Wow who would have thought no standards, free access to learning, fake-ass meritocracy, and exorbitant profits and market control would have led to such a thing
Pick a standard, whether it's a degree program, a cert, or apprenticeships. Hurry up! Just give us a solid target to aim for.
Proximate Issues
The app makers cannot easily prevent the crash since it is happening in Facebook provided code. Perhaps the only thing they could do is to code in kill switches. Another possibility would be to modify the Facebook code. That introduces maintainability issues.
The crash happened because of a server side change that triggered roughly the Objective-C equivalent of a null pointer exception.
Objective-C is an old unsafe language.
Ultimate Issues
Client-side libraries have a poor history in the industry. They are often written by server engineers who aren’t expert on the platform.
For instance, if it was really a static load method that called the server, then this was a poor design decision.
Furthermore, weaker server engineers are notorious for breaking backward compatibility. They don’t design nor build in tests to ensure backward compatibility. The backward compatibility must include backward data compatibility.
The above issues are indicative of organizational issues.
Facebook is a little bit of an outlier. It was notorious for badly written client libraries stretching back to the Facebook platform days. The joke was that their libraries may have been written by interns even though so many companies were dependent on them.
The Facebook libraries and the ilk are used for sign on through Facebook and ad attribution. The trade off an app maker needs to make it a whether to run ads on Facebook and whether to support sign in.
Facebook is not the only big company with these issues. Google’s equivalent also crashed albeit at a much lower rate due to a much more convoluted issue.
Apple introduced the Swift programming language to address the deficiencies in Objective-C. That these companies are running into these issues makes me wonder if they never transitioned to Swift. They are certainly way behind the latest version of the libraries.
To summarize, it is a cascade of issues some technical and some organizational rather than a single coding error. These issues span companies.
> Objective-C is an old unsafe language.
This was a safe crash, not an unsafe one. Swift can and would crash the same way.
Looks like the server passed empty value in JSON which got converted to NSNull in Objective-C. It's a common mistake to assume null is returned instead of NSNull.
The Swift equivalent of the JSON library should return nil which you then need to explicitly check for non nilness before using unless you use the unsafe exclamation pint.
$ swift
Welcome to Apple Swift version 5.3 (swiftlang-1200.0.16.15 clang-1200.0.22.41).
Type :help for assistance.
1> import Foundation
2> // IIRC this used to throw an exception.
3> // Now a respondsToSelector: check is silently inserted and we get a surprise nil.
4> (Int?.none as AnyObject).count + "call Swift's bluff".count
Fatal error: Unexpectedly found nil while unwrapping an Optional value: file repl.swift, line 4
2020-07-10 18:03:43.815959-0700 repl_swift[73029:930570] Fatal error: Unexpectedly found nil while unwrapping an Optional value: file repl.swift, line 4
Execution interrupted. Enter code to recover and continue.
Enter LLDB commands to investigate (type :help for assistance.)I think we are in general agreement?
This one was a good one because the app cache was poisoned so the only solution was to ask customers to reinstall the app (after Google deployed their fix, which took a while) or upload an update that nuked the SDK cache on startup.
> I will have to go and get a degree
Was not aware that having a degree made you immune to writing bugs.
Recently tested an enterprise marketing app with an SDK ... and we tested if the endpoint wasn't available. Nothing happened -- it worked just fine. Failed quietly and gracefully -- the user would never notice.
"We" know how to do this.
Facebook hasn't done anything to fix this problem since it happened a few weeks ago.
But it's Facebook -- I hope they keep it up. Can't wait for them to AOL themselves.
But if you're Facebook, and you're pushing your SDK as a great solution and it's embedded in high profile apps and you are letting this happen a couple of times, I get to make fun of you. (a dollop of sarcasm, to be sure..)
This would still happen because A) the programmers that made this mistake probably have the knowledge to pass any certification exam you can throw at them, B) most of the time we are not just building something that has exploded 1000 times before, we are creating new stuff.
The engineering process was the issue here, and sadly the only way to prevent this is not to make personal barriers of entry higher but to slow down the process with more QA, testing or whatever you can throw at it. In other words, make software more expensive. And keep in mind that this would only give better numbers but it would not solve the issue either (see Boeing 737 MAX).
Very unlikely. You probably don't realize how tough some professional exams are in some fields. We are talking dedicated months of study here.
> B) most of the time we are not just building something that has exploded 1000 times before, we are creating new stuff
Most software out there is just CRUD stuff, not really new stuff...
There's a good chance that the one who wrote this bug had a degree.
Surgeries in medicine still get botched despite all of their degrees and barriers. Degrees don't produce good code, good developers produce good code.
Please don't use uppercase for emphasis. If you want to emphasize a word or phrase, put asterisks around it and it will get italicized.
I can almost guarantee that the programmer who wrote this had a degree and passed a very rigorous testing process to get their job at Facebook. Doesn't mean anything when the company tells them to cut corners to iterate faster.
Stop blaming engineers for the mistakes of management.
“ Earlier today, a code change triggered crashes for some iOS apps using the Facebook SDK. We identified the issue quickly and resolved it. We apologize for any inconvenience.” (1)
At the scale of Spotify and other massive blue chip apps, this bug could have very well cost companies hundreds of thousands to millions of dollars in damages. This apology on the bug tracker seems so insincere. With this type of issue occurring twice in a couple months, I’ll be ensuring that no clients of mine ever install or rely on Facebook services within their mobile apps. Facebook has a responsibility to developers to ensure that their SDKs and integrated components are stable, and they have failed.
[1] https://developers.facebook.com/status/issues/17391881029111...
Facebook are big enough to blow this off, and they know it. At least... they hope so.
How is this even possible, even less acceptable from Spotify point of view?
None of the apps broken on my phone require Facebook features for core functionality. Downtime happens, but downtime due to your non-essential & user-hostile ads/tracking/social SDK? These companies deserve every single cent of lost revenue.
IMO, blame Facebook.
It's a total joke that an app would crash on startup because of this idiocy.
Turn airplane mode on, open Spotify, go to Settings > Playback, set it to Offline, then turn airplane mode off.
At least now you can listen to cached music and browse the web again.
Longer term, install a Pi-Hole on your home network and block all Facebook owned domains, because they’re a scourge on humanity and your life will be much better void of their infection.
[1] https://parachute.live/blog/forensic-investigation-the-shock...
Can you give a few examples?
> none of them had ads
The comment was about the Facebook SDK being used for ad attribution, not that the apps themselves have ads.
It creates a local VPN that sets up a local filtering DNS. I just added the following domains to the block list:
facebook.com
fbcdn.com
fbcdn.net
fbsbx.com
fb.comCheck out Blockada. (Non-root)
Adaway (root and non root phones)
Adaway requires version 5.something for non rooted devices. Anything less than version 4 is only for rooted devices.
[0] https://play.google.com/store/apps/details?id=app.greyshirts...
But you have to keep an eye on it because it can get killed when Android(?) thinks it needs the resources for something else. The app has some options to help prevent this but it doesn't always seem to work.
Also you can configure a non standard DNS (like adguard, which can block some things at the DNS level).
EDIT: or NextDNS, since it comes with some additional features.
Netguard by the same dev behind Xpirvacy, which provides dummy data to apps that try to send back all sorts of unique identifiers to the mothership (+apps like facebook).
Most popular apps call facebook IPs at launch, along with other analytics services. It's quite disgusting.
(or make your own, takes 30 seconds)
Open source and free. Don't be confused by the VPN service they also offer for a price. You just want the local firewall functionality.
Even ignoring the privacy concerns, this is unjustifiable.
So Apple have put out an incentive to their third party Devs while still giving some freedom.
Obviously it would require app developers to implement, but Apple has a lot of power here.
1) Download NextDNS from App Store
2) Open app. Hit (...)
3) Enable Custom Configuration, Config ID 22776a
It’ll DNS block some ads and Facebook domains. Turn off after you’re done. Also assume any DNS server offered by random dude as poisoned.
Edit: Issue resolved! People, turn it off or switch to your own configuration.
Mobile phones were a big step forward from plain old landline ones. But in terms of ease of communication this change was the last one with non-deminishing returns. Maybe smart phones also just for browsing HN when I wait in a queue or in the bus etc.
All those social networking and communication apps? Fcuk them.
I can perfectly communicate by calling or texting or video calling someone. What exactly is the added value of those apps?
And for Communication, you never use Slack? It's useful to segment your communications by type, I don't want to give random Tinder ladies my home phone number.
Obs: I'm being sarcastic but since it's either this or developer laziness, I'm betting on tracking since I always start from the principle that the developers analysed the situation and made an explicit choice.
in our case, it's just a wrong decision, we don't use it for tracking.
also, if you use facebook login, the terms and conditions force you to use their SDK (which automatically initialises when included and does this crap).
https://developers.facebook.com/policy/ - 8.2
"Native iOS and Android apps that implement Facebook Login must use our official SDKs for login. "
In terms of the person writing the code, they can certainly push back. Being asked to add the Facebook SDK is either a red flag enough for him/her to do so, or they don't mind. In either case, they've made that choice.
True, you're just facilitating, not profiting.
Such a clear and obvious good reason to drop support and walk away.
developers.facebook.com/policy/ 8.2 (which states “Native IOS and Android apps that implement Facebook Login must use our official SDKs for login.” This has a help icon next to it that opens a dialog on their site that shows a couple pictures and then explains “Android apps should use the default login behavior defined by the SDK, which may use the web-view Login dialog. On iOS, only kiosk apps may use a web-view Login dialog.”
sometimes i wonder if you people lost all the connections with the reality... or if you were ever employed.
I've also refused to include Facebook in my life even though the majority of my extended family practically live there and rarely communicate outside of FB posts. My wife and I just get text messages on the rare occasions instead.
Yes, been employed and unemployed. And I'm under no illusions that FB login support will get dropped anytime soon. I can still lobby for the desirable alternative.
Someone recommended airplane mode and lo and behold the app started working. This is Facebook's fault, but also Spotify's.
How such an established platform with supposedly-established engineers can let a third party library crash their app at startup is genuinely embarrassing.
If Facebook, which charges money for advertising, let go of controlling that tracking relationship, they’d lose a lot of money. So they won’t.
Not again...
This is what FB for dev says:
"To make your app compatible with the latest iOS, be sure to use the latest Facebook SDKs for iOS."
:)
Anything that can open the general public's eyes to how much data is leaked through these kind of practices is a good thing. Hope it amounts to some non-negligible loss for Facebook, Spotify and the others.
The simple act of listening to music should not require a vast server infrastructure at the other end of the world, under control of a third party, and a working global communications network. The fact that people got used to this kind of nonsense is very, very scary to me.
The act of listening to music doesnt require a vast server infrastructure. Thats why people could still listen to music they had stored offline during the outage.
Forgets selected play list from previous day. Every single day.
Sometimes would play to wrong audio output, which would produce the same silence as first problem but restarting/ rebooting wouldn’t fix. Finally I’d realize and switch it back in control panel.
I’m surprised most popular apps haven’t learnt their lesson since this happened last month as well.
And then reject new app updates which don’t follow the new rules.
Or just outright ban this spyware.
Especially if SDKs are statically linked.
Don't allow the files to install.
Block all communications.
Ban it completely except in the Facebook authored apps.
Q. E. D.
facebook.com
fbcdn.com
fbcdn.net
fbsbx.com
fb.com
instagram.comMild inconvenience for now, hopefully this will have more people move away from the FB SDK.
Thanks, send her the github issue.
The crash is:
Crash information
-[NSNull count]: unrecognized selector sent to instance 0x1cd6fe1e0 2020-07-10 17:24:26.052924+0800 OSDemo[4198:604792] * Terminating app due to uncaught exception 'NSInvalidArgumentException', reason: '-[NSNull count]: unrecognized selector sent to instance 0x1cd6fe1e0'
Any idea why seemingly no one of the big apps does this?
Like wtf. they have so many resources how is it possible that even with that they can't make their sdk error resilient.
Maybe the mass revocation of EV certs happening ~today is the trigger?
"HackerLemon is listening to Band on Spotify"
I would vote for "not use it" but others want Facebook integration supported and happily share information on all their users to Facebook ...
P.S. I believe part of this is also Apple, who don't like runtime loading of code, as that makes their verification harder, but I'm no iOS dev or user.