Google employees are listening to Google Home conversations

The difference between 3 and 4, noble as it is, can be caused by feasability concerns that push people into 3, not just ignorance of the privacy impact. Human labelling of training data sets is a big thing in supervised learning. Methods that dispense with this would be valuable for purely economic reasons beyond privacy - the cost of human labelling of data samples. Yet we don't have them!

Techniques like federated learning or differential privacy can train models on opaque (encrypted or unavailable) data. This is nice, but they assume too much: that the data is already validated and analyzed. In real life modelling problems, one starts with an exploratory data analysis, the first step being looking at data samples. Opaque encrypted datasets also stop ML engineers from doing error analysis (look at your errors to better target model/dataset improvements) which is an even bigger issue, IMO, as error analysis is crucial when iterating on a model.

Even for an already productivized model, one has to do maintenance work like checking for concept drift, which I can't see how to do on an opaque dataset.

One thing that I think gets lost in engineers (and humans) is scale.

Googazon doing {thing} might be "meh" for 10 people. But the implications look very different when it's doing {thing} for 10%+ of a country's population.

At 10 people, I may find out Ted likes to eat Italian. At 10%, I may find out an Italian chain has a sudden health issue and short their stock.

Which is in essence their original playbook: do things that only work at a scale that only we can play at.

It doesn't matter if a company takes your data, does a poor job of anonymizing it, and then decides to label it as "training" for their "AI" or if they just stick it all in a flat .txt files and process it in Fortran.

It's the exact same thing. You should be mad about the data being saved and used. Splitting hairs over implementation details to try and find a loophole is just a waste of time.

Yes I'm frequently amazed how many coworkers I have that are still completely plugged into Google, Facebook, Amazon services/spyware, fill their homes with internet enabled "smart devices", have alexa/google assistance etc, and yet they act like I'm paranoid when I try to discuss security concerns or just flat out don't care.

As much as I hate to say it, I think there needs to be a massive breach or abuse of power from one of these organizations/services that has severe real world consequences for those that utilize/support them. Until then nothing will change.

I continually come back to the same thought: The Amish (to take a random example) consume probably thousands of times less than I do. They do without the conveniences of technology which I "enjoy". They have neither computers nor an endless stream of distracting media and social interactions.

Yet they are probably, on average, as happy as or happier than other Americans.

These technologies don't really exist to serve people - they exist to create profit, and they do that by meeting superficial needs.

There is great potential for machine learning to help the human race. But the vast majority of energy dedicated to the question by the information megaopolies isn't invested in that direction except under the most facile sorts of hypercapitalist justifications.

We should, in short, regulate data collection and analysis into oblivion, with exceptions made only for democratically determined use cases which serve the public good.

I am sure we can whip up some such AI easily with some quantum computing. Preferably in a blockchain so we can verify correct operation and scale better.

ducks

"Mycroft uses opt-in privacy. This means we will only record what you say to Mycroft with your explicit permission. Don’t want us to record your voice? No problem! If you’d like us to help Mycroft become more accurate, you can opt in to have your voice anonymously recorded."

(project is open source, at https://mycroft.ai/)

Let people participate in R&D if they want to, but don't force it.

Asserting your point of view as "educated" and "correct" while labeling people who don't share it as dangerous. Doesn't sound like a great way to start a discussion.

I think this goes well beyond tech workers. I think it's time for society to legally recognize the balance between the value of ML systems and the privacy concerns of customers of ML.

Doctors and lawyers obviously should understand the value of privacy, but we, as a society, have also created legal rights and duties for them. Conversations with lawyers and doctors are legally privileged; at the same time, there are specific consequences for medical companies or lawyers who do not protect that information.

Companies like Google, Apple, Amazon, etc. certainly have the resources, intelligence, and sophistication to comply with a similar regulatory regime. IMO it should be possible to construct a law that allows companies to collect, store, and tag customer data for purposes of training ML systems, but sets serious duties, with consequences, on them to do it right.

Right now, what is to keep employees at these companies from abusing these systems to stalk, to surveil, to harass, or even just to feed their own curiosity? These data systems are core trade secrets for these companies, which means they are opaque to any kind of oversight from outside the company.

The free market can't create the necessary balance because customers need information to make decisions--information that they don't have. The result will be an increasingly chaotic "hero/shithead rollercoaster" as customers make snap judgments based on scanty or wrong information about what these companies are actually doing.

This is a classic case for regulation, which prevents a "race to the bottom" of sketchy practices for short term gain, while also protecting the ability of people and companies to use this technology to create value.

Doing this right will help data-leveraging companies in the long run, just like attorney-client privilege and HIPAA have helped lawyers and doctors build trust (and therefore value) in their customer relationships.

And the first one to do it should be apple itself.

Do you think its possible to be educated on the societal implications of these technologies and still not be concerned? Seems like you've written your own viewpoint into the only "logical" one here.

There is absolutely no proof of number 2 in your list, but that is by far the widest-held belief.

It's infuriating, because we can't have a useful societal dialog about the issue if the largest chunk of concerned people are, essentially, conspiracy theorist.

TBH its not the Googles I'm worried about. They're under a fair amount of scrutiny, but ultimately, I think its mostly chicken little screaming regarding the 'how' of how they'll use this data. The ones I'd be more concerned about are the ones not on your radar. Google knows security, anonymity, they have share-holders, governments know to regulate them, etc.

What you really have to worry about is the companies getting data on you with out your permission that you have ZERO knowledge they are doing so, and as well, almost no ability to know they are taking data on you.

The reality is, most people are almost completely ignorant of how much real world data on them is out in the wild with no google or facebook or any tech giant involved. I can't get into too much detail on it, but the data is there. You just have to be savvy enough to find it, process it, predict, and refine.

In terms of morality? I don't know.

No one is afraid of power when it's in their own hands. A common failure mode is that people assume a given power that's in their hands today will always be.

Ok, I agree completely with you, 100%. However, based on my limited worldview, tech workers barely understand the tech they work with at all [0]. Asking for the ethical implications to be mulled over is unlikely to happen considering the near-weekly HN threads on "interviewing sucks, heres how to fix it, lol". We can't even figure out how to hire someone let alone how to impedance-match with them on deep issues like ethical implications of ML/AI.

[0] https://stackoverflow.com/

I think that issue is much bigger than just training ML based off real world data.

Laws might give you some recourse if you can prove this is happening or not. But really we need to start thinking about ways we can ensure the entire stack is under the users control.

I have said this in the past. Free software and open sourc movements have worked decades to open up software and hardware for greater transparency and user control. Yet within a few years we have wiped that out with all the closed hardware and software we happily hold in out hands and place in our homes.

We need more open source hardware and software projects offering alternatives to these systems that have become a daily part of our lifes.

The law needs to catch up. Sharing should require specific informed consent and legislation needs to establish a scope where data stored as a "tenant" on a third party server is given 4th amendment protection.

Google using the data to train models is just a tool, it's a baby step, they aren't doing that to sell the models or in and of itself, they're doing it so that they can generate data that they might consider theirs and not yours from your voice data and then feed that in to other systems which generate tremendous profits for them in ways you don't even know. They have intended uses already. Is it a remotely fair question to talk about ethical training in this context without some idea as to the intended use and distribution of the meta data?

If the training already only happens on a 1/500 sample, skewing the sample towards "people who don't care about their privacy" will probably not significantly impact the quality of the data.

I'm surprised this wasn't already the case, but hopefully the article will help the people responsible make better decisions in the trade-off between minimizing onboarding friction and respecting user's privacy in the future.

I fall into that category as I have time, nor do I trust any evaluation methods, to determine if a company is using my data ethically. If I create data and store it something that’s not mine, then I only do that in situations where I’m comfortable with the owner doing anything they want with it.

I understand ML and know that Google has to at least use it for training. I’ve also worked on IT long enough that even in super tight controlled environments data are misused by administrators.

Knowing what I know about how people I have worked with have come close to or have actually mishandled data despite the best of intentions, I do not trust any of these teams without an explicit accountability mechanism that is observable by an outside entity. I'm not looking to punish slip-ups, because mistakes happen, but I am looking for external enforcement to keep people honest.

It's not that I think the engineers using this data are mustache twirling villains, it's that I think mishandling is inevitable due to inattention (yes, even you make mistakes!), and we have to design our data pipelines against that.

4 because not being explicit about the practice is misleading at best, outsourcing the difficult task of keeping the analysis private show how unimportant it's considered, and because big techs have a tendency to decrease privacy over time. Using clients who paid for the product as a dataset generator is also wrong.

But 3 at the same time because well, it's important to evaluate the performance of the product in the field not just in the lab. There were so many cases of catastrophic failures for ML models (ex. classifying black people as gorilla) that having a tight feedback loop is important.

It has to be done right, but evaluating a product that was primarily developed for (or at least by) English speakers and transfered to other domains seem like the right thing to do.

All in all, I don't and wouldn't use one of those assistants because 4 outweigh 3, but it's not binary.

If they're collecting it in a way that can be requested by governments (for instance) or could be leaked by hackers that's another layer of valid "concern" not related to my understanding of the ML aspect of this.

The more I see of this tech, the more disturbed I am about its existence.

In some ways, dystopia and utopia really are the same thing.

agent( tech, management ) # assuming management has power over tech worker

understanding-of-ML( yes, no )

concerned-about-ethics-and-privacy( yes, no )

The below combinations are worst in terms of ethics.

{ agent[tech], understanding-of-ML[yes], concerned-about-ethics-and-privacy[no] }

{ agent[management], understanding-of-ML[no], concerned-about-ethics-and-privacy[no] }

{ tech[management], understanding-of-ML[no], concerned-about-ethics-and-privacy[no] }

So my thoughts are like #4, but with the undertone of "who am I kidding, of course they don't care, what can I do?".

This is perhaps the most dangerous response.

Isn't there a law in some US states that there needs to be consent before recording someone? How does this fit in and who would be held responsible, the owner of the listening device or the company behind it?

I really don't understand how they don't allow setting a custom hotword.

Google says "the excerpts are not linked to personally identifiable information." To me that means the metadata is stripped, not that they strip anything out of the audio.

Even more fun, if you call a bank, you often have to key-in your account number (which can be easily decoded if your phone sounds back the tones, which most do), then tell you name, your address and sometimes your other PII like Social Security number or part of it. Record that call and that's a complete identity theft package, nicely wrapped, just replay it to the bank (which name you've also have recorded, if the user called on speaker, which they did because who wants to keep the phone pressed to your head all the time while you're waiting and listening to the muzak) and you get full access to the user's bank account.

Meanwhile, Android allows you to personalize voice commands based on its ability to recognize that a specific person is the one saying "OK Google". Voice authentication has already reached high accuracy with a few seconds of unconstrained text, or a few words of fixed text. Voice identification on open sets takes more data, but sub-minute clips are still reasonably effective.

At the very least, Google itself could make a credible attempt to identify whether the speaker in any voice clip heard by Google Home is a regular user, and plausibly de-anonymize users of OK Google. More alarmingly, we're told that about 1 in 500 Google Home clips is heard by a human, and this employee apparently shared "thousands" of clips with a news organization. It seems plausible that anyone with access to any large voiceprint database could attempt to obtain clips from a random contractor and de-anonymize the most interesting or salacious content.

It’s hard not to feel like this outrage is trumped-up anti-Google FUD. So many more worthy fronts to assail Google et al. on!

After all, they let you upload photos and video that are, per various policies and with some non-zero frequency, reviewed by humans — and users are begging them to do it more often.

I'm not familiar with EU law around these things, but I would imagine there is some kind of whistleblower mechanism available, and a right for authorities to audit/inspect such activities?

Also how is having access to small samples of audio a privacy issue? Are they also receiving enough information to attach an identity to the audio clips? How long are the clips? Are they randomly assigned to humans? Do those humans get to listen to multiple clips from the same Home device and can they tell that's the case?

[1] https://en.wikipedia.org/wiki/IBM_and_the_Holocaust

“Once men turned their thinking over to machines in the hope that this would set them free. But that only permitted other men with machines to enslave them.”

Frank captures the motivation, the fact that it is desirable is the scary part.

Akin to the genetic engineering scene from Gattaca.

People listening to what is said to a voice assistant after the prompt doesn't really alarm me. What would be alarming is if the voice assistant was secretly recording normal conversations.

After all, the voice recognition in my Prius doesn't send recordings of my "call mom and dad mobile" back to Toyota and it certainly doesn't send random snippets of conversations my wife and I have to low paid contractors around the world.

> The man, who wants to remain anonymous, works for an international company hired by Google. His job is to listen to the audio clips and to write out what he hears, so that Google can improve the speech assistant.

So,

> It is quite another thing if they are selling my conversations to third-party vendors.

That is exactly what Google is doing.

2. Sounds like you're okay with Google using your private communication to enhance its products. So they can target you with ads based on your conversations with your spouse, children, or colleagues.

3. I just hope the algorithms understand context:

Your spouse: son, put down the gun and come to dinner. [jokingly] Oh no, don't shoot! Argh you killed me, now eat your peas.

You: we're going to kill it this quarter. They'll never know what hit them.

TV in background: "kept their body chained in a basement. That and more at 11"

Siri: O__o

As a user, I expect to see feedback declaring that it's having trouble and specifically requesting permission to send my unparseable requests to some review queue. I expect to see feedback indicating what could resolve the issue: was it something I said? was there too much noise in the background? was there a software fault and a fix will be deployed at such-an-such date?

The fact that I do not get any feedback whatsoever when things go wrong leads me to have no faith that the problem will be resolved in any satisfactory manner. When combined with a complete lack of consumer/business interaction options in general (w.r.t. Google), it leads to some very dissatisfied consumers.

You think engineers are flagging voice requests that happen more than once? How would engineers have access to that data in the first place if it's supposed to be anonymous and private?

No, its only creepy if you're not sure whether or not conversations had in the "privacy" of your own home are being listened to by a faceless person

Regardless of how the software works, it's still incredibly creepy.

It would be nice if companies like Google made it very clear that specific forms of user-data can/will be human reviewed for development/operational purposes. Personally, I just assume that anything I say after activating a digital assistant will be anonymized but listened to by a human one day. And I still think that Digital Home Assistants are pretty awesome despite that.

Unsupervised learning is totally a thing! Word embedding models are achieving STOA results on massive unlabeled corpuses. That's what's powering most of the new results in NLP.

Clustering and Dimensionality Reduction folks just sitting here thrown out to dry too...

"Google would like to use the last 15 minutes of voice data to improve Google Home. This may contain sensitive information. Do you approve?"

Install these devices in the homes of google employees, executives and offices and allow the public to listen in. What’s good for the goose is good for the gander and all.

Maybe when google has trained the systems enough to not need to train them by collecting and listening to customers conversations, then they enter them into the stream of commerce.

Operationally, though, here are a few thoughts:

- Ensure that the persons doing this review do not have physical access to the systems outside of a secured environment (one in which outside audio and video recording devices are not allowed, and for whose presence is monitored, with physical access controls, etc.) Basically, not remote workers or a typical office environment. Most finserv call centers do this, so it's not particularly crazy to think they can do the same.

- Mask the voices such that they are intelligible but not identifiable. Maintain a limited set of "high access" taggers who can hear the raw clips if there is an issue with the masking.

- Limit the length of the clips (sounds like they already do this).

- Have pre-filters for anything personally identifiable in the audio. The metadata for the audio might already be de-identified, but what if the audio clip consists of the person reading out a phone number, credit card number, username, etc.? They should have their "high access" team building detectors for that and flag those portions of the audio or whole clips and route them to a limited access team.

- Make it more clear to customers that their audio, including accidental captures, can and will be sent to their servers. Make this very explicit, rather than burying it in TOS and using terms like "audio data". "The device may accidentally interpret your bondage safe word as a trigger and send your private conversations to our real-live human tagging team for review."

- Provide a physical switch that can temporarily disable the audio recording capability.

- Pay money, like cigarette companies do, to help fund a public education campaign that informs the general public about these listening bugs and mass surveillance issues so that people are aware of industry practices and how it affects them.

Edit:

I like what others are saying about explicit opt-in, as well as paid end-users. For quality/safety control, I don't know that they can exclusively use paid end-users. They probably need to sample some amount of their real live data. For that, explicit opt-in makes sense.

That's because we, as an industry, have fooled them into thinking that AI is real, and that people who don't know it's real are idiots. We don't think students deserve a proper tech education, so unless they are professionally techies, they have to learn from marketing materials (designed to convince them to buy things.)

I mean, we've told them that we're on the verge of AI (and to fear it), and we can tell a two year old to do something and they do it without having to hire somebody to help them understand us.

Another serious question: couldn't you ask ME to fix the data every so often (like the "improve this traduction" feature of google translate) ?

Which isn't to say it isn't frustrating, just that your frustration at your friend is misplaced - they're not exposing you to anything which you aren't already exposed. If your fear is that these tech companies are going to misuse casually-overheard conversations from these devices, they're just as likely to use it from all the phones you're near.

If your fear is that your friend is going to use this tech, however, maybe you need better friends. There's nothing stopping an unscrupulous friend from putting plain-old audio recorders through their house anyway.

My guess would be that the person who does consent to the terms of service of the device and plugs it in will be the one responsible for ensuring that it is used in a way that does not invade the privacy of anyone else. It would be similar to me buying an IP camera, agreeing to the terms of service then putting it in my neighbours house without their knowledge.

In the context of the GDPR, I think the question comes down to what rights a third party person has when a user puts their personal information on the company's servers. If someone uploads a video of me without my consent to YouTube, can I get it removed under the GDPR?

Click on Voice & Audio Activity, then Manage Activity. You can listen to and delete every recording Google has of your voice. It's a little trippy to be honest. They've had this since long before the GDPR.

Also, how do you explain the progress on ImageNet challenge over the last 10 years? Each method there uses the same dataset, yet error rates (top5) have gone from 30% in 2011 to around 3% in 2016.