Facebook Patent Imagines Triggering a Phone’s Mic When Hidden Signal Plays on TV | Hacker News Reader

Facebook Patent Imagines Triggering a Phone’s Mic When Hidden Signal Plays on TV

gizmodo.com

255 pointsmmaanniisshh88 comments

If anyone wants to try ultrasonic transmission, I have a demo that runs in your browser, and boy do I love to plug it

https://quiet.github.io/quiet-js

Very cool, glad you mentioned it!

This is pretty amazing, worked well out-of-the-box between my laptop and phone. Makes me think of all the weird ways that one could exfiltrate data (to use a different example I've seen, convincing a device to puke its firmware out via an LED)

Also I'm pleasantly surprised by my ability to hear (barely) the 19kHz carrier.

Some group either got caught or proved you could exfiltrate data via the fans on the air gapped computer to the microphone on the connected one. It was obviously incredibly low speed but possible.

I honestly don't recall if this was just a trial or it was used in the field by a spy agency

I saw one where they got audio out of a silent video of sound waves in the area vibrating a potato chip packet on the ground.

http://news.mit.edu/2014/algorithm-recovers-speech-from-vibr...

That was impressive work, but they do require a camera that can record at 2000-6000 fps.

Check out the last portion of the video. They got something out of a 60fps camera, although it's far from good quality.

That's logical. Nyquist won't budge just because your idea is clever.

https://en.wikipedia.org/wiki/Nyquist_rate

They actually got Nyquist to budge to some extent. Check out the last portion of the video.

They took advantage of the fact that digital cameras tend to scan line by line over a short period of time, so there's actually more timing information available in an image than the base 60fps.

It looks like you might be thinking of this:

https://www.wired.com/2016/06/clever-attack-uses-sound-compu...

https://www.wired.com/story/air-gap-researcher-mordechai-gur...

I've seen a prototype IoT thing (let's say a lightbulb) that paired with its peers by transmitting a public key through a flashing LED. Never became a product, but made me realize how easy such a thing is.

I've heard of this being used in a few products, especially more advanced "LiFi" systems. I think one company was advertising an in store network for things like setting price signs and such based on optical data transmission using special bulbs.

A friend owns a few "smart power strips", you connect them to a wifi network by installing a phone app, then when you're ready holding the phone screen up against a light sensor on the strip. The phone screen flashes a bunch to send the network info and whatever else over. It seemed like a non-techie friendly way of doing it while also being cheaper than either adding a bluetooth module and having the phone connect over that first, a central hub, or a little screen + keypad like printers.

This is brilliant in its simplicity and implementation, your awesome!

That's awesome. Wonder how hard it would be to do it steganographically and embed it in existing audio.

Basically, you can’t really use an additive method for that purpose.

Bluray has a watermarking technology[1] that works by modulating the existing sounds in the content. It transmits _very_ slowly, but the result is moderately imperceptible to the typical person. It’s also fairly durable against distorion.

1: https://en.m.wikipedia.org/wiki/Cinavia

What is a rough estimation of the rate for wireless mode? I saw the 40 kbps figure for the cable mode, but nothing for the mic+speaker case. Yeah I know it would greatly depend on the hardware and the distance, but still, I'd like to know some ballpark out of curiosity.

Do you think this could be used to automate transmission of whatever information needed to pair two phones? E.g. to establish an ad-hoc wifi connection with the intention of sending some files (a la Lenovo's SHAREit, direct transfer without requiring internet connectivity), so the user doesn't even need to read some secret code from one device and write them in the other.

I did some experiments for pairing devices over WebRTC via sound [1]. The link has a proof-of-concept page that you can try to pair devices and transmit files over LAN.

[1] https://github.com/ggerganov/wave-share

Thanks that's also very interesting! Listening to how an SDP Offer sounds like, was amazing!

Depends quite a bit on your setup. I put together an interactive sandbox for testing modem profiles. The short answer is about a few kbps for close proximity.

https://quiet.github.io/quiet-profile-lab

Thanks, tomorrow I'll be checking this out for sure :)

Very cool! Though on my MBP at 50% volume the ultrasonic transmission is clearly audible. Although much less audible than the other profile.

This is apparently already in use: https://arstechnica.com/tech-policy/2015/11/beware-of-ads-th...

Yeah it's SilverPush isn't it, I remember this from a few years ago.

No matter how many ways I come up with to do things I'd never actually do there always seems to be someone who both thinks of things I didn't and absolutely would.

Depressing really.

Someone has a dataset of every (?) apk from the Play Store - they need to scan for SilverPush and release the results.

Here you go!

8 apps with SilverPush currently integrated: https://mixrank.com/playstore/namespaces/com.silverpush.sdk....

68 apps that have removed it: https://mixrank.com/playstore/namespaces/com.silverpush.sdk....

The FTC cracked down on SilverPush a couple years ago, which explains why almost all apps have removed it: https://www.ftc.gov/news-events/press-releases/2016/03/ftc-i...

I noticed recently that a YouTube ad that played on my device had a high pitched whine in it. Could have been an audio editing error, but I wondered if it was some kind of trigger, perhaps to see if I am near others when it goes off.

I really dislike being tracked. I wish it was easier to “go dark” and still own a smartphone like device.

You and me both.

Check out light phone 2. Might suit you.

I’m still waiting on my book reward from their first campaign. Am not giving them any money until I get what I paid for.

It would be nice to capture that and decode.

I wonder if you could solder a low pass filter into the electronics that drive the speakers in a TV in order to disable this sort of tracking. It might be even easier if you have an external receiver.

A lot of fingerprinting technologies use something called Difference of Gaussian (a type of bandpass filter). However, in practice you capture interest points across multiple bands (scale space).

Basically what this means is that interest points are in multiple frequency bands.

A high or low pass filter is unlikely to filter out a significant number of these interest points at other frequencies so the fingerprint is still able to match with a fairly high confidence.

If the fingerprint frequencies are in the audible range, wouldn't you hear a brief humming when the fingerprint signal is being broadcast?

I've been thinking the same.

Another option might be jamming with ultrasonic white noise, but I'm not sure about the health & comfort effects of having something like that going 24/7.

Google, Amazon, and others who have speech-based "smart" devices don't even need this. Just apply Shazam to commercials. I would be very surprised to learn that they weren't already doing this.

Just apply Shazam to commercials. I would be very surprised to learn that they weren't already doing this.

Shazam does this on both commercials and TV shows.

I've seen several times "Shazam this ad for more information!" at the bottom of the screen.

I think peterlk might be talking about a more sinister version where the computation is performed without consent.

With all the Facebook 'We're sorry' on TV it's a no brainier. They want to see if you're still surfing Facebook while they apologise. Also they can see if the money their spending is effective.

This is a patent, no one has actually made it, and big corporations file tons of patents just to have them. It's fine to keep an eye on it, but it's not a big story because, fundamentally, no one has done anything or has concrete plans to do it.

As bad as Facebook is about watching your every move, don't forget that Gizmodo, as part of Univision, is also a large corporation incentivized to push your buttons for money.

Probably naive of me, but I also see a potential silver lining in large companies patenting evil ideas: it's a very effective mechanism for ensuring they don't get commercialized. And if they are, you know that FB is either behind it or has condoned it, so can focus your efforts to stop it on one target.

Again, potentially anyway.

> it's a very effective mechanism for ensuring they don't get commercialized.

False. Patents can be licensed and anyone with money can use it. Also, the patents are more like legal ammo which is used only when needed. So it's actually a mechanism for ensuring that:

- nobody who bothers the large company commercializes it.

- nobody but the wealthy members of a 'cartel' commercialize it.

> This is a patent, no one has actually made it,

False. This has been in use by ad companies for quite a while now. One implementation, Silverpush, was reverse-engineered three years ago: https://www.theregister.co.uk/2015/11/20/silverpush_soundwav...

And the company advertised it's services a year before that: https://techcrunch.com/2014/07/24/silverpush-audio-beacons/

More sources: https://arstechnica.com/tech-policy/2015/11/beware-of-ads-th...

https://thehackernews.com/2017/05/ultrasonic-tracking-signal...

https://www.wired.com/2016/11/block-ultrasonic-signals-didnt...

https://arstechnica.com/information-technology/2017/05/there...

Fair enough, though I meant "no one at Facebook".

Indeed, facebook most likely uses a variation of the method by Silverpush, to subvert the copyright.

After all, they're known to use the microphone, but it's not known what they use it for:

https://www.computerworld.com/article/3079412/security/faceb...

https://www.independent.co.uk/life-style/gadgets-and-tech/ne...

Not even a patent. Just an application.

Right, because Facebook gets denied for patents all the time.

Your sarcasm is misplaced. Big companies don’t get special treatment from the USPTO.

Or the FDA, or Justice Department, right?

Seriously. Regardless of reason, how many patents does Facebook get denied for?

> and big corporations file tons of patents just to have them.

and more often than not the patents are extracted from either something that's already working in production or from a proof-of-concept. I realize most corporations are printing money but patent applications are definitely not cheap. Nobody puts money behind an application without having something palpable.

> This is a patent

No, it is not yet a patent. It is a publication of an application for patent.

If you look the publication up in the USPTO's public pair system (https://portal.uspto.gov/pair/PublicPair) you will find that the last activity was FB's attorney responding to a USPTO rejection on April 12, 2018.

Isn’t filing a patent “doing something” about an idea? This is more than a mere daydream from inside Mr. Zuckerberg’s mind.

It absolutely can be, but my impression is that it's usually a way for engineers to pad their resumes and corporations to accumulate patents to pad their IP assets. In support of your case, advertising is very much in line with Facebook's business model, so it's a bit less speculative than I was characterizing it.

Silverpush, an Indian company, was already doing this 3 years ago:

https://www.forbes.com/sites/thomasbrewster/2016/03/21/silve...

Couldn’t be happier with the decision to delete my Facebook account.

Be careful next time you buy or shackle yourself to a new phone, it will probably have an unremovable FB app. I see no reason why it would need you to have an account to track and analyze you.

This is why it's so important that open source forks of Android are available. Without them, it's not possible to have a modern mobile communication device that you can actually trust. (Arguably even then, since drivers tend to be opaque blobs and the radio subsystem runs its own closed code, but at least you're no longer free real-estate for any application big enough to pay for an unremovable app.)

Not if it’s an iPhone.

Agreed. Apple respecting user privacy at some level is a big decision factor for me lately.

Don't worry Google, Apple, et al just listen to you outright.

I agree it’s hard to really stop being tracked but I’ve been trying to become less of an easy target lately. Stopped using gmail as my main email and made duckduckgo my default search engine. Plus the traditional browser extensions.

Far from perfect? Yes. But at least I take some confort from no longer being such a juicy ad target.

You may have made yourself a juicer target by doing that, though…

Could you elaborate? I understand the concept of a you-shaped-hole-in-the-graph, when you opt out. But is that what you’re referring to?

It’s kind of like putting your stuff in a safe rather than showing it to everyone: sure, nobody gets to see what you put in. But it’s probably something of value.

It's like knowing that you are a duckduckgogo user/or that you don't like being tracked is also a valid data point for advertisers.

I remember seeing a post yesterday on HN where they mentioned that some ad networks treat the 'Do not Track' preference as an additional data point about your personality instead of simply honoring your preference.

Apple? There’s no indication whatsoever that they listen to anyone.

Hey Siri! Are you listening to me?

(I know, it's more nuanced than that, but the "my phone is a always-on microphone" isn't accurate either)

That’s just straight up misinformation.

A phone listening for a key phrase simply is not ‘Apple listening to you’. There is no nuance about that.

It's not misinformation, and there is more nuance to that than you think.

Your phone listening to you to say "Hey Siri" is an Apple device listening to you. Yes, as for any reasonable implementation of an assistant, that detection and recognition is done on the device - but there's the issue of false positives, i.e. your phone can mistake a part of your conversation for the keyword, and start listening to you and upload said conversation. Then it is quite nuanced as what counts as "Apple is listening to you".

I understand what you are saying, but it’s neither accurate nor reasonable.

For one thing, it is false to say ‘upload said conversation’. It’s true that a false positive can lead to a sound sample being uploaded to Siri.

That is not uploading a conversation. Saying a conversation is being uploaded is clearly mischaracterizing what is happening. Why exaggerate from a sound sample into a conversation, which is a quite different concept unless your goal is to mislead?

Separately, the uploading of short, misidentified sound samples does not constitute ‘Apple is listening to you’. This is also clear.

In any public space, people around you may briefly give you some attention if you say their name, even if you happen to be referring to someone else with the same name.

Only paranoid people construe this to mean that everyone is listening to them all the time.

To accept your reasoning that ‘Apple is listening to you’, we must also conclude that if you are in public, ‘everyone around you is listening to you’, rather than the regular understanding which is that people pay attention to you when you say something that is relevant to them.

You can decide for yourself what counts as 'Apple Listening to you', but this particular reasoning seems a lot like paranoia rather than what most people understand 'listening' to mean.

If you say 'Apple is listening to you' based on exaggeration and uncommon definitions, unless you explain what these are, most people, if that believe you, will interpret this using the common understanding and be misinformed.

Saying 'Apple is listening to you' is therefore misinformation.

From what I understand, this isn't a problem for users on iOS who haven't given microphone access to the offending apps right?

Would also like to know this. And even if you did allow access, there would be the big red recording bar shown at the top of your screen when activated, right?

This is my understanding as well. Even if something has permissions granted to it by default as the system apps do, they’ll still show usage.

Nope, just tried this https://quiet.github.io/quiet-js on safari on my iphone. It just asks for permission, after that there is no indication that it's listening or not

Tried the same. It has red microphone indicator in the URL bar.

Safari, iOS 11.4

An app can't listen to the mic unless you grant mic permission, unless the app has been bestowed with magic entitlements by Apple. For example, Apple Clips is an app in the app store with magic entitlements that bypass cam&mic permission prompts. I have no idea why they brought out the big guns for that. Also, Uber was running with a magic screen scraping entitlement for a while, apparently to support Apple Watch in the early days.

Neilsen has a similar API. Trigger your phone, send coordinate data when inside a geofence. Sure it's for a better user experience.

did i misunderstand the fact that it needs the microphone on listening for the signal to turn the microphone on to record another signal?

That was my question as well, we must be missing something.

Oh good. It's the CueCat again.[0]

I've been meaning to do a blog post explaining why this wasn't the technological disaster everyone makes it out to be. Perhaps this is an opportunity.

[0]https://en.wikipedia.org/wiki/CueCat

Yet another piece of previous art: the Spanish La Liga recording from the phone app to identify unlicensed broadcasts: https://news.ycombinator.com/item?id=17296449

yes, but facebook "cares about user's privacy" and people here on HN will stick their head firmly into the sand and swear up and down that this is somehow excusable.

violating privacy is their business model. there are no (zero) limits to the lengths they will go to fulfill the end of violating your privacy whether or not you are a user of their service. i highly doubt any user would knowingly consent to this.

You can never get a free lunch. If you ain't paying for it, you're paying for it with your data.

This is a little bit outdated. Nowadays you pay with both your money and your data. For example, see Windows telemetry. It even collects the full command line for applications elevated through UAC.

But it's not a privacy violation, because you accepted the TOS and it clearly states in section 57, paragraph 22, on the 15th line: "Facebook application may use the microphone to enhance your experience." /s

Can someone please explain to me why, if this is possible, I have to point my remote control to the TV to make it do something?

"Imagines"

Reply on news.ycombinator.com