Facebook confirms that it tracks mouse movements
indiatoday.in
indiatoday.in
More interesting is "a patent held by the company states that the Facebook app uses voice recognition algorithm, which uses audio recorded by the microphones, to modify the ranking scores of stories in users News Feed." and their speculation that Facebook could soon reveal details about their use of surreptitiously recorded user audio.
Facebook makes a curiously specific denial about audio, which is that it is not used for advertising. Considering their entire business is basically advertising, what does that leave? But all they mean is ad selection. When they were found to be recording audio during the posting of statuses, I believe they claimed it it was so they could recognize the music you were listening to, and know something about your mood. So for a long time, I have thought that they use audio to select other content, like friend suggestions, or to inform the selection of stories that appear on your newsfeed.
Correction: Not in the background, but it could do it silently while the app is in the foreground.
I can only assume that up until P developers could record both sound and video in the background. That's some scary shit.
[edit] Why isn't there a system level visual indication or audit trail of any camera or mic access? Surely this would be trivial to implement, you could even disable it if you just didn't give a toot about privacy at all.
I wonder if the apps and browsers will get a "grant access for 5 minutes" setting, so that one can feel safe with the services of these services. EDIT: Or maybe lock-screen/pulldown-screen (or whatever it's called) notification that has a "remove permission" button so that you can remove it when you're done.
[Disclaimer: I work on Firefox.]
In Firefox you can grant the permissions once (for the current page only) or grant them forever, your choice. If your browser doesn't offer that choice, that's a problem with your browser.
[Disclaimer: I work on Firefox.]
However, I think that I disagree with you on whether or not sharing the data is important. If you are heat mapping me, like facebook and probably everyone else from Microsoft to CNN and FoxNews does, or you are recording me like everyone from Facebook to Samsung does, I'm sorry, I've got a problem with it. I don't care if you don't share that data. I don't want Samsung recording what's going on in my living room. Doesn't matter if the data isn't shared. It's just the principle of the thing.
It's gotten to the point where I actually purchased a certain model of Sony TV, because the teardown verified that there is no microphone in it. Then I tossed the remote control and got a generic remote with no voice control.
People joke about me being paranoid, but I'm not paranoid. Sheez... I'm old and boring, I know that no one cares about what's going on in my house or on my computers.
I'm just stubborn.
Why let the privacy invaders win?
Edit: I wanted to add that I didn't intent to focus on whether the data is shared. I think FB having and using it is bad enough, especially if they're ubiquitous. Also, once anyone creates such data, other entities such as governments will seek to obtain it and likely do so eventually.
And, since we can't know in advance all the ways data can be combined, recombined, projected, and analyzed there is no such thing as informed consent to use said data unless specifically restricted to a single analysis using only given data.
Such as, any time you store a precise time in connection with user actions that has privacy implications. I picture simply not recording the time or exact URL. If the system is designed without any sort of privacy in mind, and just records whatever data is convenient and too much, that's easier to abuse than one that intentionally records a minimum with privacy in mind. I agree it's amazing the way all of this can be subverted, and yes, I realize that HN is stocked with data scientists who are more knowledgeable about this than I am.
I got carried away and rambling but I mean, come on. Mouse movements? Really? I suppose you have to give them credit they are creative in a very perverted sense.
You need to be careful though because the microphone is in the remote control instead. So buy the remote control WITHOUT voice control, get rid of the remote that comes with the TV, and you're good. There will be no mic's in your tv setup. (This will cost you a bit extra. About 20 or 30 bucks for a generic one.)
A second problem is that it is a smart tv, so you have to make sure to NOT connect it to the internet and get your smart services via a box you control. (I haven't been able to prove that Sony siphons smart data usage from its tv's the way the Samsung Android tv's do, but it is technically possible for them to do so.)
In any case, under no circumstances should you get a samsung. You can't even use those tvs without the privacy invasion equipment activated.
>You can't even use those tvs without the privacy invasion equipment activated.
Could you elaborate further? If I don't connect a samsung smart TV to my network, shouldn't I still be able to use it as a normal TV?
Indeed, I would never buy a Samsung smart TV (or phone).
I don't do anything im worried about people finding out, but that doesn't mean I want to let them listen in either. Maybe if someone was upfront about it from the beginning or showed us what they were truly doing, but seeing how shady it all seems to be makes me assume they are doing shady shit with it.
One bad thing about this system for Android is how much control Google has over permissions - for example, their own built-in Shazam...
Does the patent really state that Facebook does that, or Facebook spammed the patent office with obvious ideas about how they could do that. Big tech companies have loads of trivial patents on stuff they have no firm plans to build, just to stake out IP territory.
> I believe they claimed it it was so they could recognize the music you were listening to, and know something about your mood.
When did this happen, and why wasn't it frontpage on HN and all the news sites?
As far as the status audio, I'm sure it has been discussed on HN. I don't have the time to dig up all the info right now but here is FB's take on that: https://newsroom.fb.com/news/2014/05/a-new-optional-way-to-s...
The media and general public seem to be 10-15 years behind when it comes to understanding how the things they rely on, and the tools being used to “improve” them, work
Though IMO, a lot of the blame is on Facebook and the whole lot for avoiding discussing openly in order to avoid fallout. Just asking customers too is out of the question, of course. Cause BIG Corps are smarter than their customers
Only once you’re “too big to fail” can you be honest about your shady BS
I'm looking forward to technical solutions to verifiably disable and prevent this kind of tracking.
To those talking about "tracking UI usage": do a UX study. Sit some people down and watch how they use the site. Ask them questions about what works and what doesn't. Stop spying. You got all this damn money and you can't be bothered to actually lift a finger to do some difficult work that involves interacting with all those "dirty" people out there. FFS most people would probably be happy to fill out a survey if it actually would impact the product in a positive way. Creepiness is creepy.
All this leads me to think mouse movements tracking is much more widespread.
However I'm pretty sure this was advertised or at least acknowledged by Google in the launching of reCAPTCHA v2.
https://www.wired.com/2014/12/google-one-click-recaptcha/
> IP addresses and cookies provide evidence that the user is the same friendly human Google remembers from elsewhere on the Web. And Shet says even the tiny movements a user’s mouse makes as it hovers and approaches a checkbox can help reveal an automated bot.
[1] From Book "Silence on the Wire"
I agree that this is something people should be more aware of and school is a good place to start. However it is up to browser manufacturers to fix this, not users.
Proof: https://panopticlick.eff.org
However, most biometrics aren't the best single line of verification. You still have to add a backup verification of some kind.
Examples, I was working on my car all weekend and my hands have new calluses on them and my finger print is off. I got a new keyboard and my typing pattern is different, etc.
(You can always open the browser inspector and check network traffic for each page or, if you are using Chrome, dive into chrome://net-internals/ )
Stupid cat and mouse game. How difficult would it be for a bot to simulate a human's mouse movements? I suppose not very difficult.
Also, doesn't this conflict with rare types of input devices? Or people with a motor function disability?
> to also determine if the window is foregrounded or backgrounded
Shouldn't there be an API for that?
As for the latter, iirc there are blur/focus(-like) events for the window object. Maybe mouse movements gives them better confidence? Because of course you want to make absolutely sure your users are seeing all the ads.
http://idlewords.com/talks/website_obesity.htm An interesting read on this cat and mouse game.
How difficult would it be for a bot
to simulate a human's mouse movements?
And even if the bot author had all that information, it would still be super hard to write an AI that accomplishes a given task in a way that mimicks a human successfully. It would mean to win a 'mouse turing test'.
Shouldn't there be an API for that?
This issue touches on the real privacy problem the net is facing. It's not the wrong cookies or privacy policies. It's fingerprinting. There is no technical solution to it.
Don't forget that Facebook's false positive rate should be very low. There are lots of humans on their platform, and they should all pass the test.
This makes it easier to construct a bot that will pass the test.
I can imagine it would be easy to trick the system a few times (either as a bot pretending to be human, or a human acting like a bot), but tricking it consistently over months or years is going to be damn near impossible.
I've only given it a few minutes thought, but position and time data is really small, and easy to compress (you don't need to send anything while the user isn't moving the mouse). If it's sent in batches or over an already open websocket, it's not like it's using a ton of resources on the client.
Assuming all of their users (guessing a billion daily active users) are on desktop half of the time (a wildly incorrect assumption I'm sure), and the mouse position data is 1mb per person for the data you care about (which again, seems like a lot), that's 500tb.
For $25k you could store it all. That's nothing compared to the benefits of being able to identify bots on your platform.
Using these metrics you could probably start to draw some characteristics of how your mouse acts based on what you are doing and where you are moving your mouse.
This could then probably be used to build some form of algorithm that moves the mouse for you with noise (accelerating up & down along the way, deviation from a straight line, stopping in the middle of the line, etc.).
At least the mouse movements themselves shouldn't be difficult to do given a source of data. Simulating that you click on same FB UI elements as real people with same statistical properties on other hand is where you might be lacking the data to do it properly
This is a cartoon version of deep learning.
You need to break ReCaptcha? Simple, you implement your own captcha on your own site that's frequently used and whenever you need to solve one you copy the challenge and present it to one of your users.
Same with recording mouse data.
It's an old idea even, very similar to https://xkcd.com/792/
You don't need to learn from all humans. You need to learn from very few (or just even one).
Not all problems are machine learning problem.
But does Facebook track user during everyday session or just during some validation-action?
There is one : https://developer.mozilla.org/en-US/docs/Web/API/Page_Visibi...
Although I think it can be a privacy concern and somewhat of an anti-feature. For instance, Youtube uses that API to stop playback on mobile when the page or browser is not in the foreground. Of course, there's an extension for that ...
I also suppose not very difficult in principle (imitate any nearby human's movements and Facebook should not complain), but it is not within the focus of a general bot developer and therefore makes the whole project exponentially more difficult.
I suspect this is part of covering their ass for GDPR. Pose everything as a security problem, so you can claim you have legit interest in tracking all of that.
> How difficult would it be for a bot to simulate a human's mouse movements?
Simulating? Extremely difficult. Perturbing pre-recorded paths is a bit easier to do, but requires pre-recording of a lot of paths. One of the fastest ways to get your Poker bot banned is to not fix this one way or the other.
> Also, doesn't this conflict with rare types of input devices? Or people with a motor function disability?
It still tracks the mouse movements, just now being able to classify their users as disabled or using an arcane device (both of which are interesting tidbits to add to your advertisement profile).
If I'm interested in the content of a page, I either swipe the mouse off to the edge of the screen or put it on an area of whitespace so I can scroll with the scroll wheel.
The thought that people are hovering their pointers over stuff they're actively looking at strikes me as odd. Oh well.
Being able to aggregate data or inspect individual sessions is a useful tool to learn how users navigate with a site.
Keyboard keystrokes get captured too but the systems are intelligent enough to filter out passwords and payment details.
I don't really like this form of monitoring either but I've seen it in several companies.
Sure. Except they are not.
Citation needed. But i did not realize that before, so thank you very much for this information, i will desactivate js on every page with a password field from now on.
"Following the recent report that Mixpanel, a popular analytics provider, had been inadvertently collecting passwords that users typed into websites, we took a deeper look. While Mixpanel characterized it as a “bug, plain and simple” — one that it had fixed — we found that:
- Mixpanel continues to grab passwords on some sites, even with the patched version of its code.
- The problem is not limited to Mixpanel; also affected are session replay scripts, which we revealed earlier to be scooping up various other types of sensitive information.
- There is no foolproof way for these third party scripts to prevent password collection, given their intended functionality. In some cases, password collection happens due to extremely subtle interactions between code from different entities."
https://freedom-to-tinker.com/2018/02/26/no-boundaries-for-c...
Our own prototype, from before I've actively joined that particular project, tested on some live website helpfully displayed all the content of some textarea of some request form that somebody started to fill in, but afterwards decided not to include some of the details. That was a big eye-opener for me that it's absolutely not a right thing to do. We have ended up implementing a debounced indicator "some typing activity is occurring right now on this field", but we still had to deal with feature requests about content collection.
Judging from quality of some of those competing solutions, I certainly wouldn't bet that they're "intelligent enough". Maybe in 75% of the most common cases, maybe.
https://freedom-to-tinker.com/2017/11/15/no-boundaries-exfil...
So, it's for spying on users. What happens then?
It would also help out Facebook if someone sent them a daily minute-by-minute log of what I was doing. That doesn't mean they should go try and do it.
A modern social network is a spy agency where you file your own reports on yourself in return for being able to read those of other people.
It's a very anti-privacy environment we live in right now. But it's for our own good or so we are told.
Yes
Centralised social networks are by nature spy networks in that they collect a huge amount of data on a large group of people, both data concerning those people as well as data relating those people to any of the others. As to whether a centralised social network uses these data for nefarious purposes or only to provide a service to the users is up to those currently in control of the network. Ownership can change, a once-benign social network can turn into a nightmare overnight as the data will be there for the new management to exploit.
To be fair, this makes sense from a UX standpoint. You don't want messages to be marked as read if you have your window open but have walked away from the computer.
Do they use some type of client-side library that caches data for a while and asynchronously uploads it occasionally? Or occasionally try to asynchronously sample the mouse position and just get a coarser set of data?
It seems like real-time requests that respond to mouse changes would create huge performance problems and/or be easily stopped with browser extensions.
Personally, I find this even more disturbing. Does this "only" apply to TVs where a user is logged on, or are they also building shadow profiles for any smart TV that comes with the Facebook App preloaded and is connected to the internet?
At least for Samsung and Sony, I can easily see them cooperating with Facebook for a negligible fee.
"But it's not nefarious..."
"But everybody else is doing it too..."
"It's not for surveillance, profiling and shadow profiling, pinky promise, trust us..."
It's also an efficient way to distinguish people from machines. Think about the click-only reCaptcha for example.
Would you change mouse each time you change browser tabs, keeping a mouse for Facebook and a mouse for other websites?
Also, keep in mind mouse tracking is just another technique deployed to identify you. It's not supposed to be a silver bullet.
I absolutely not believe that. for real. this is something i keep reading but a seriously not see how mouse movement can be unique from one person to another. it's not a fingerprint come on.
> It's also an efficient way to distinguish people from machines.
well that i can believe, but not more
It's a lot easier to get one company (or person) to stop a practice only they do than it is to get them to stop doing something that everyone else does too.
Facebook has ties to and influence over government, it is so big and far reaching that I feel it's right to be more concerned about FB doing something like this than other smaller players.
The scale of FB is what makes it a special case.
However, I think that the proper approach to something like this is educating users. Companies are gonna capture your mouse movements, it's not something we should legislate over, but users should be informed as to what it means to give companies like Facebook information about yourself.
Worrying about mouse movements when you freely send clear text messages to their data pile about your most intimate feelings and thoughts is ass-backwards.
Going after the largest offenders and making a big splash is more effective than doing nothing until you can get every company to simultaneously stop something.
No. Not only does facebook not get cover from "everyone else is doing it," they are actively perpetuating that cover existing for others. They are one of a few entities with the weight to set "industry standard" simply by changing their practices.
>while you can get up in arms about it, you need to get up in arms about the practice industry-wide
I am.
In other news, Google Tag Manager and crazyegg exist.
They said very specifically they don't record what you say for advertising. It was specific enough that it almost seems an admission that they use it for something else.
However, there are plausible explanations for your spouse getting ads for things she's spoken with her phone in the room - for example, the patient googled the medication later or wrote about it on facebook, and facebook knows thay your spouse and the patient likely had a conversation based on location data showing that they walked down a hallway together.
Furthermore, I don't believe that Google and Apple (well.. less Google, more Apple) are in cahoots with Facebook to give them a backdoor to device permissions.
That's funny, because very recently, there was a discovery that fb gave most of the device makers deep access to user data, including Apple.
Against all evidences, Apple still gets a pass.
What they mean by deep access is that the Facebook sharing system extension built into some OSes (including iOS pre iOS 9) had the possibility of accessing a lot of information from the connected Facebook account, which is not really anything to worry about (if your device’s OS is malicious you have way more to worry about).
We’ve used inspectlet from time to time to help figure out where there are problems in our UI. It tracks mouse movements as part of a complete session. It’s been really helpful.
What's the easiest way to hack your browser to give no information or dummy information here?
There's also the difference between announcing "hey, look at this cool tech we have to make the web faster!" vs "we are legally required to admit that we've been watching you like a hawk, for... reasons."
It's a very manual process, but probably one of the most powerful tools for improving user experience I've ever seen. And typically for most businesses, you are keeping these sessions for 2 weeks or thirty days at most.
But neither of us know Facebook's policy (although we can easily guess their intentions, based on past behavior), and nobody ought to be expected to cut them any slack.
Quick search indicates the tracking information starts on page 84.
Page 86: "Device operations: information about operations and behaviors performed on the device, such as whether a window is foregrounded or backgrounded, or mouse movements (which can help distinguish humans from bots)."
We need a new science of hysterias. The internet provides ample data for robust analysis and prediction. Any takers?
"But Facebook is using it to signal advertising!"
Well. Everything you do on Facebook is used to signal advertising. I thought we were kind of all aware that was what happens with this free service. /shrug
document.body.addEventListener('mouseenter', e => {
console.log('this just in, erickj tracks mouse movements');
})