Hacker News Reader

Comment by minimaxir | Hacker News Reader

It's worth nothing that the reason security researchers haven't just intercepted the traffic from the Facebook apps to see if its transmitting voice data is because the apps use Certificate Pinning, which prevents the SSL traffic from being decrypted using the SSL certificate generated by mitmproxy/Charles.

In light of that restriction, what might be interesting is looking at the amount of data transferred by the Facebook app with/without the microphone/location services enabled. (this is a data project I have in the pipeline)

View on HN

Note that while certificate pinning does make reverse-engineering harder, it's also a legitimate security feature; without it, anyone who controlled a CA (including most major governments) would be able to forge a certificate and use it to spy on users.

Certificate pinning is a hurdle to reverse engineering, but a surmountable one, at least on Android. Since the app is running on a phone where you may potentially have root, you can pick it apart with a debugger and see the traffic before it leaves the phone. This is technically challenging, but it is something that people do sometimes.

Or you can find and replace CA file/string with your own and then do mitm. There is virtually no way app developers can defend against such hacks on any platform owned by user (so, everywhere except on Apple devices).

Would something like this work as a method to authenticate authentic clients?

http://www.cnn.com/TECH/computing/9908/20/aolbug.idg/index.h...

Paraphrasing the article, it would be for the server to use undefined behavior in the _authentic_ clients to determine that they were in fact authentic. In this case, a buffer overflow doesn't appear to crash the client, but lets the server know that it's talking to a legitimate client. That's quite clever.

On jailbroken devices such an effort would be about as trivial.

It's more annoying than you'd think if your adversary anticipates this and designs around it. Fragments of the CA can be used within the codebase, effectively making you need to pull everything apart and with security canaries and embedded interpreted scripts even if you're getting close you never know if you're still black and if you're still seeing the same thing as everyone else.

Well, it turns into a reverse engineering arms race at that point. There's no way they can guarantee anything, but they can throw resources at obfuscating things more.

If you're the reverse engineer and I'm the app author, you find/replace my CA file. Then I respond (or anticipate!) by checksumming the file to detect tampering. Then you respond by find/replace on the checksum.

Then I obfuscate the checksum string. Then you respond by faking out the platform's checksum API so that it always returns true. Then I respond by computing a checksum that I know should fail and verifying that the checksum API isn't just always returning true. Then you respond by faking out the platform checksum API with a whitelist of blobs whose checksum it should lie about.

Then I respond by statically linking my own checksum verification code into my binary instead of calling the platform's. Then you respond by patching my binary to jump around the code. Then I respond by using code obfuscation techniques.

And on and on. Given enough time and resources, any implementation I create can be subverted. But if I'm a huge tech company, I can afford a lot of time and resources too, if I want to. I can't eliminate it, but maybe I can make it something that rarely happens.

But if I'm a huge tech company, I can afford a lot of time and resources too, if I want to. I can't eliminate it, but maybe I can make it something that rarely happens

The whole cracking scene can afford far more time and resources than any one tech company. All adding protections does is make a more valuable target, because crackers love a good challenge.

"There's always a crack in everything. It's how the light gets in."

> But if I'm a huge tech company, I can afford a lot of time and resources too,

Can? Sure! Would?

Security is one of those things that most people say they care about, but they are really not willing to pay for. Big companies have resources, but they are also in the business of making money, so they will put most of those resources to work on features that produce a ROI. Security is a huge cost center, and even when taken seriously it will be pursued only to the degree that it addresses/mitigates risks enough to conduct business.

Not really. A skilled reverse engineer does not find/replace any certificates, or anything like that. They just debug the app, stepping over all the code, instruction by instruction. There's no way you can beat that. I'm a malware reverse engineer - I work with this kind of stuff every day. And I'm pretty sure there's no 'game': once the binary is in my disassembler, it's over.

It's funny how all the security measures for "your safety" always seem to reduce user's transparency and control.

It's an unfortunate feature of reality.

From the point of view of software, it's impossible in principle to tell whether or not the code being executed does what the user wants it to, and only what the user wants it to. Half of that is the halting problem, the other half is that "what user wants" is an General-AI-complete problem. Moreover, the software can't even tell the difference between "the user" and "a malicious third party".

In meatspace we solve this problem with rules and laws. Software, for better or worse, moves around too fast.

Not really.

I gave it a thought, and decided I don't need apps to be able to obtain highly elevated privileges, root or similar. This is, indeed, dangerous, esp. regarding ADB root access (a rogue "charger" + an accidental wrong tap[1] = totally compromised device that can be only fixed by full re-flashing). I needed my own firmware that does things my way, signed with the keys I control.

So I did. Now all the "secure" apps are happy, and I still have the control over my device's behavior.

( Okay, I've cheated - I had to sanitize androidboot.verifiedbootstate when kernel initializes, because I can't control the bootloader :( )

[1] Hm, maybe password-authenticated root access is okay, though... But not a typical "tap to allow" dialog.

It's the same problem posed by child-proof lids of drug bottles.

I think Android has features for detecting whether the device is rooted. Facebook could then disable spying, in effect Volkswagening the situation

Magisk (Android root utility) has the ability to hide itself very well. Google Pay and most, if not all, banking apps are not able to detect it. It is a cat and mouse game for sure, but so far the Magisk developers are keeping ahead of that curve.

Wasn't the most active development on Magisk from when Pokemon Go was really big? I remember there being an arms race with Niantic at some point.

If I’m not mistaken, it is common for windows malware to act differently if it is running inside a VM, to avoid analysis.

Yeah, then you'd be <a href=https://www.urbandictionary.com/define.php?term=munsoned>mun...

It's very feasible to reverse engineer the Android APK for Facebook and replace the cert with your own.

https://serializethoughts.com/2016/08/18/bypassing-ssl-pinni...

http://blog.dewhurstsecurity.com/2015/11/10/mobile-security-...

Facebook already has plenty of issues with people not trusting them. Lying about this and then being caught red handed would be devastating from a public opinion perspective. Even if you assume a completely amoral team that cares about nothing but ad revenue, do you genuinely think they’d risk something as dangerous as that? Especially given how much you can still achieve with the data that is known to be collected?

in the current climate, getting caught doing anything just means you have to slime your way out of it, rename/rebadge the activity and then carry on when the attention goes somewhere else.

is uber still the most hated company or has the magnifying glass moved onto somewhere else?

What makes you think that user trust isn’t one of the things they’re willing to break in order to move fast?

I've met a lot of people outside HN who believe FB is listening to their conversations through the mic. I haven't yet met one who has stopped using it as a result.

I think you wildly overestimate how angry users get about privacy violations. For examples, Target, Yahoo, Home Depot, and Equifax have not been screamed into rubble.

To be fair, people who believe that are not bright bulbs. They fail at very simple statistical analysis and conspiracy theory testing.

To be fair, almost everybody fails at statistical analysis, even scientists whose work depends on it.

Many of them haven't stopped using facebook, but many have uninstalled the app.

How devastating would it be, really?

It would likely be a blip in the news, and then people would move on, as usual.

Remember the Sony rootkit fiasco? People still buy Sony, and most people probably either never heard of that incident, don't remember, or don't care. Buying whatever the new Sony gizmo of the day is is more important to them.

Microsoft has had endless spyware fiascos, and people still routinely buy Windows, as long as they can play their games or run Office, that's all that matters to most of them.

Then there have been scandals like Enron, where the execs knew that they were doing something that was clearly illegal, and that their company really would be devastated if what they did was ever revealed. These "smartest people in the room" did it anyway.

Corporate history is full of just such deceptive and destructive practices. I'm not sure I'd put Facebook above that sort of thing, a priori.

Exactly. People would be outraged! and they will happily log into Facebook to post their outrage about Facebook.

I remember when some news broke about Facebook doing something or other a year ago and the common response seemed to be to make an official sounding wall post saying "I do not give Facebook the right to do X with my data", sort of like Michael Scott yelling "I declare bankruptcy!"

I can see that happening again with voice recordings.

Speaking of badware: last week I found Mac Afee installed on the laptop of a relative that I manage. I assured me that he had not installed it himself (I had not installed anything actually). Could it have come from windows updates ?

if you manage their device there's a great chance they have no fucking idea what they're doing and could easily have installed it unintentionally

No, Windows Update doesn't distribute third party software. And even if it did, MSE is technically a competitor to McAfee so there wouldn't really be a solid reason to distribute it.

Your relative probably installed something and pressed "Next" through all the dialogs including the ones asking if they want to install super helpful bundled software.

McAfee comes packaged in the installation of some other software, e.g., Java, where it can easily be installed by mistake if you aren't paying attention.

True. At the same time, there are pretty solid rules against recording the audio of someone's conversation that would be unambiguously illegal and allow for actual prosecution.

But it's an interesting question: if someone credibly proved that FB was "wiretapping" on such a massive scale, would they get prosecuted? How much could they do in their own defense? Are they so enmeshed that prosecutors wouldn't bother?

Feels like a case of "unstoppable force meets immovable object".

Any app that had microphone permission at all already has your permission to use your microphone.

Uber broke almost every moral rule and still everybody's using it.

I refuse to ever install the Uber app, specifically because of their conduct.

So not literally everyone... but still many.

I can see someone having to install the uber app... but having to install the facebook app? just use the web app... that might be possible with uber too but I'm not sure because I don't use uber

m.uber.com works fine without installing anything.

The Uber app was "helpfully" pre-installed by Microsoft on my Lumia 950.

It was the second to go, just after Facebook.

> being caught red handed would be devastating from a public opinion perspective

For what it's worth, this public opinion backlash has not appeared with other companies: "Of course we're not working on leaked project [x]". "Look at project [x] we're working on!"

Even Facebook's own under-disclosed psychological experiments have been largely forgotten; Facebook has suffered few if any long-term ill effects from it.

The psychological experiments scandal was a nothingburger from the start. Facebook is not an academic research institution. They already operate with user's consent. Every business that engages in advertising or product design is performing psychological experiments.

Their customers love it, and with customers I mean advertisers. For users to leave there must be somewhere to leave to. And there needs to be another business model that does not depend on marketing. Maybe selling actual hardware, with built in social media.

Hubris is powerful. Look at the dumb behavior exhibited by Uber as an example.

How much worse would being caught lying be compared to straight up admitting it? I'd say it's worth the risk for them.

"which prevents the SSL traffic from being decrypted using the SSL certificate generated by mitmproxy/Charles"

There are some shady companies out there [1] that use the mic to listen to what shows are being played real time. [TVs in the US are on all the time] (Check out their customer list). These companies need this pinning.

[1] https://www.audiblemagic.com/

Couldn't someone just patch the binary to accept a different certificate? Or for that matter, read the decompiled bytecode and figure out if the app is listening? Or run a patched Android that logs all microphone API calls?

I suspect I'm overlooking something, as surely some security researcher would have done some of this already.

Yup, patching the binary works fine.

Unless that package/binary is signed - which Android ones are.

Then patch the signing...

...and if the signing code is also signed, then patch that.

It's patches all the way down. ;-)

Certificate pinning is relatively easy to disable on jailbroken iOS devices and rooted Android phones. I'm not highly technical and I did it myself to sniff Snapchat traffic 2 years ago. Any half competent security researcher should be able to do this trivially.

While certificate pinning is annoying, it's just an obstacle not a roadblock. I haven't met a security research that mentioned this to be a prohibitive feature.

There are couple ways out, from revers engineering the binaries, through jailbroken/rooted phones, running apps in simulators, etc.

You can't beat certificate pinning with offline analysis?

You can, if you want to read reverse engineered code. Takes more time, but completely doable. I do that on all Android apps.

Facebook recording voice data and transmitting to their servers in the US is not only highly illegal in most parts of the world it's also unacceptable to basically everyone. At some point it would have been leaked internally.

The idea that Facebook is doing this is just ridiculous.

Surely no large organaization has ever been found to be doing illegal or unwelcome things.

Quite sure that listening to the conversations of billions of people is on a different scale than what you typically see.

I suppose you don’t see things of this scale often but I would argue that big things like this are happening all the time. I was watching recently some interviews from 2005 about the justification for the US invading Iraq in 2003, and the disparity between what the public was told about WMDs and what the security community believed was maybe on a similar scale (depending on how you view privacy and war). And then the people who made decisions that led to the 2008 financial collapse were doing things that hurt people on what you may consider a similar scale, depending on how you view robbing people of their wealth versus robbing them of their privacy.

Facebook invading our privacy by recording persons of interest or the people en masse would be horrible and in many ways unprecedented, but it wouldn’t be beyond the levels of abuse we have seen from powerful people in the past. I can easily imagine that the app supports hot mic capabilities and that they do turn it on sometimes at least at the request of law enforcement. And then the question is... when else would they turn it on? And would that program ever grow? Would they ever fork the program so each team involved thinks there working on a small project? This is all speculation but I can imagine a situation where it starts small and then grows until it seems like an insane program but everyone involved is accustomed to it.

^ actually, IMHO the fact that you assume that "The idea that Facebook is doing this is just ridiculous." is what is ridiculous.

Some companies do "highly illegal" things all of the time. It all comes down to the fact that whoever is in charge:

1) doesn't hold to a moral system that restrains them from doing said illegal things (or at least doesn't hold to one consistently)

and

2) thinks they can do said illegal things without getting caught, or if they are caught, thinks they'll be able to recover reasonably well from any punishment (if there is any) that is handed down.

Personally, I do not know whether Facebook is recording/transmitting data like this, but I guess if I found out they were, I would not be surprised -- given the things the company has done in the past, and the things their leadership has said and done in the past.

I'm reminded of this: https://en.wikiquote.org/wiki/Mark_Zuckerberg

What if they just analyze the audio locally for keywords, and then transmit that?

Most of the analysis done about this controversy boils down to Facebook not listening because they can buy it otherwise obtain data that is more effective.

When you read their responses to this controversy, pay more attention to what they don’t say. Whomever aggregates your viewing habits by listening to audio from your TV may be listening, for example, Facebook just Hoovers up the data.

Leaked internally... to Facebook employees...

If the speech recognition is done on the device, the amount of data could be very small indeed. If I was to do this, what I would do would be to defer the heavy lifting of speech recognition (if it caused any significant load on any of the chips) and perform it in the background, later, when the device is known to be plugged in to power. That way this kind of work would not be impacting the battery. I don't think looking at the amount of data is going to tell you much unless they have done things in a spectacularly stupid way.

Speech recognition is quite compute-intensive and the load of running it on the device would be obviously detectable - not necessarily by random users, but definitely by any security researcher who'd care to do so.

Also, storing the data until it's plugged in would require an unusually large amount of storage, and that would be detectable.

I was replying to a commenter who was proposing to inspect what was being sent over the network, if you'll just read what they said.

That aside, speech recognition isn't that heavy of a process these days if all you're looking to do is extract keywords. We used to do industry-leading large vocabulary continuous speech recognition on a Pentium 133... phones these days are way beyond that without breaking a sweat. Detectable? Sure. But remember, I was talking about this person's plan to look at network data.

Furthermore you don't need to store all data. You can store only when the phone is hearing stuff, as determined by a super lightweight measure of magnitude that does no speech recognition whatsoever. Is the storage detectable? Sure. But again, what was I responding to? Network traffic monitoring.

You could also look at the variance in amount in different environments for different sensors (noisy/not noisy), (complex image/flat color).

Is there any reason that pinning should not be optional?

In other words, any reason that the user should not be able to "disable" it? (Use own root CA.)

Consider that in this case, the data being transferred to Facebook belongs to the user.

Is it unreasonable for a user to require that they be able to see what data is being transferred before they agree to transfer it?

Enterprises do this on PC --- they have to be able to MITM all their user's traffic. Not sure how mobile devices are covered.

As for transparency, look to GDPR and friends to see what rights are being declared.

Considering that certificate pinning protects hacked user devices from making insecure communications, it would completely defeat the point if the user could disable it.

I don't think so. Certificate pinning protects from rouge CAs.

Wouldn't it be possible to crack an app (decompile etc) to disable Certificate Pinning? What comes to my mind also is a debug to obtain TLS master-secret then one can decrypt encrypted stream post-factum (Wireshark accepts master-secrets).

Yes, and there are tools to do exactly that. https://github.com/sensepost/objection

You could vary the complexity to encode the sounds you input (silence,pure tones, white noise, psychocoustically weighted white noise, etc.) to prove that that they're transmitting them.

Different (esp. older) versions might be another variable that would be of interest to a researcher.

why not inspect the traffic locally before it gets encrypted?

Chrome is moving away from certificate pinning [1].

Even if it's an issue, a security researcher could recompile Chromium or Firefox with certificate pinning turned off and test with that.

[1] https://groups.google.com/a/chromium.org/d/msg/blink-dev/he9...

The browsers are moving away from certificate pinning. The apps aren't, as they control both ends. If I made a Facebook replacement today i'd do it too.

Reply on news.ycombinator.com