Apple contractors listened to 1k Siri recordings per shift: former employee

This is just how assisted machine learning works. The model needs to be updated with corrected annotated data that its gotten wrong in the past. It needs to have enough of this data until the model accounts for it properly so that it begins to prioritize the corrected label instead of the incorrect one.

You're not going to be able to go in and hand-fix the model so that it doesn't cause these errors. This is one of the biggest ongoing issues with convolutional neural networks - you cannot debug them to determine why something went wrong.

>If you want to improve your voice recognition, then you need to have a ground truth

This is true, but that doesn't automatically make it okay for them to offload this expensive resource to their customers without making it transparent that they were doing so.

>This is literally what google voice was for

Who cares? It's completely expected from Google, they use their customer data to do everything. I expect more from Apple. They have a (so far) great track record of protecting their customer's privacy. I fully expect Google, Facebook, et al to be doing ML on my photos, and that's why I'm bought into the Apple ecosystem instead of Google's. I value that my data is private more than being bombarded with stupid "collections" that the Google Assistant puts together with AI.

I really hope this doesn't indicate a new direction for Apple.

Why not just let the users give feedback on the responses? I’ve always been frustrated that I can’t.

(Edit: before downvoting this post, or any others on HN, please consider reading the whole thing to make sure you actually disagree with it.)

I agree that it shouldn't be surprising, but what does surprise me is that this isn't a massive legal problem for Apple at the criminal level. Every time someone uses Siri in a one-party consent state, isn't that a violation?

Seems like everything could have been fixed with a EULA clause saying that Siri queries and all other communications with Apple are "recorded for quality purposes."

The fact that we aren't talking criminal consequences suggests that there is, in fact, such a clause in there somewhere, and that this entire story has been grossly sensationalized.

I'm trying to write a polite response to this, but this just comes off as so cynical to me it's hard not to be outraged that this is the top post in this thread right now. What's to stop you from apply the same argument to medicine? Why shouldn't we test the side effects of new drugs by testing it on people who don't know they're being drugged? We don't just want to improve AI, we also want to improve exercise equipment and parcel delivery. So of course you just have to surreptitiously record people's workout routines and put hidden tacking devices inside the products they buy off amazon. People should just know that if you buy a fancy Japanese toilet with a built-in bidet and heated seats, that it's also streaming video of your anus to HQ for quality control purposes.

Just because you want to improve your product, that doesn't mean you can do anything you want. No one should have to assume that the products they're buying are Trojan horses that are secretly spying on them.

>I don't see how any of this is surprising.

Had thay said it before hand, I am pretty sure 99.9% of Apple's user would trust Apple and just click yes.

The problem is they didn't ask, nor did they tell. What privacy entails, in Steve Jobs's own words

Privacy means people know what they’re signing up for, in plain English, and repeatedly. I’m an optimist; I believe people are smart, and some people want to share more data than other people do. Ask them. Ask them every time.

Make them tell you to stop asking them if they get tired of your asking them. Let them know precisely what you’re going to do with their data.

> don’t see how any of this is surprising

Agree, not a surprise, and already on the record. Here, from 2017:

Siri records your queries too, but she doesn’t catalog them or provide access to the running list of requests. You can’t listen to your history of Siri interactions in Apple’s app universe.

While Apple logs and stores Siri queries, they’re tied to a random string of numbers for each user instead of an Apple ID or email address. Apple deletes the association between those queries and those numerical codes after six months. Your Amazon and Google histories, on the other hand, stay there until you decide to delete them.

http://themillenniumreport.com/2017/03/not-only-are-alexa-si...

From Wired, “Apple finally reveals how long Siri keeps your data”, in 2013:

Once the voice recording is six months old, Apple "disassociates" your user number from the clip, deleting the number from the voice file. But it keeps these disassociated files for up to 18 more months for testing and product improvement purposes.

"Apple may keep anonymized Siri data for up to two years," Muller says "If a user turns Siri off, both identifiers are deleted immediately along with any associated data."

https://www.wired.com/2013/04/siri-two-years/

In general, customers are used to this pattern: “This call may be recorded for quality assurance purposes” clearly means someone will be listening to it later. The same “product improvement purposes” language for voice assist would naturally mean the same thing.

As a very minor point of fact, Google did have a service that (allegedly) existed primarily to collect voice training data, but it was GOOG-411, not Google Voice. Google Voice still exists, is an excellent service, and to my knowledge has not been credibly accused of primarily serving as a data collection tool.

It is surprising to anyone who doesn't work in machine learning.

Personally, I remember Google telling me that they "may use recordings to improve the voice recognition quality" or something like that. What I did not realize is that that meant humans would be listening. Sure, any ML expert will say "Well obviously! A machine can't train itself!", and in retrospect I wonder why this was not obvious to me. But here I am, a software engineer who worked for Google for eight years, and it was not obvious to me, somehow.

These services need to state more clearly how data is being used. When you sign up, it needs to say, clearly: "Human employees of [company] may listen to your recordings for the purpose of improving voice recognition."

I think the reason we are not presented with such clear language is because the people building these products know they'll lose a lot of users. But that's no excuse to mislead people.

I am surprised that Apple was doing it without asking for permission. I would happily allow Apple engineers to listen to failed interactions if I had a chance to pick and choose which ones to give them access to.

I personally don’t have a problem either way, but there is a nuanced difference.

If someone presses the home button to actively activate Siri they are purposefully sending messages to Apple and shouldn’t be surprised that the data is being processed and maybe even listened to by humans.

However, I could see someone being slightly more concerned about accidental activations causing data to be sent to Apple.

It would also be easy for Apple not to allow humans to listen to recordings triggered by Hey Siri.

I find it surprising, I would have assumed Apple would simply used subset of informed and paid users to get new data. Just because you need to improve the system does not mean you need to use the recordings from everyone.

>Now, ofcourse apple have positioned them selves as a privacy first company, so this is a big no no.

I don't think this is a "no no" based on Apple's privacy stance. The terms do say that they can send the recording in order to perform additional analysis and there's no user data, user ID info, or anything else that would give up a user's privacy included in what's sent over. Unless someone managed to recognize a specific voice in a recording, there's literally no way to tie a specific user to that recording because Apple anonymizes all of it on-device before it gets sent out.

The issue really isn't that they are doing it. It's that they were literally advertising the opposite.

https://9to5mac.com/2019/01/05/apple-privacy-billboard-vegas...

I went to that link they provided in the billboard, and then followed it to here.

https://www.apple.com/privacy/approach-to-privacy/

Nothing under the Siri and Dictation section says anything about letting contractors listen to your audio.

The kicker here isn't that Apple was listening to these recordings. The issue is they advertised something else, and then on top of that, Apple sent recordings to 3rd party contractors.

Why the outcry when Microsoft did the same though?

Of course it makes sense to do this if you want to improve your training set.

>This is literally what google voice was for, it was a way of google getting lots and lots of samples of people talking using a phone's microphone.

That's what GOOG411 was for as well.

> I use Android phones, and there are many times I wish there was a human listening

Google does the exact same thing to train the “OK Google” functionality. Apple wasn’t mechanical turk-ing Siri, this is about humans tagging prerecorded data for ML training datasets.

I've learned that you can say "Feedback" and Google Home will let you dictate some feedback.

I don't know if anyone is actually listening, or if it's just being thrown into a huge dataset somewhere, but it's something.

Recently GHome broke reminders for me. I'm still upset about it.

You opt out by not using the service, which is what I've done since the beginning.

You can't opt out anyway, everyone has a smartphone and many have Siri and other services activated. If Siri didn't occasionally activate by accident in meetings I wouldn't even be really aware of that.

I work for a company where we have contractors at the customer service centers. This is pretty standard for customer service workers. They handle reservations over the phone and customer support. The centers where they work are very locked down. They deal with customer names and credit cards as part of their jobs. They go through background checks. (It can be hard to get enough people who both apply and can pass the checks ) They cannot bring in phones, memory sticks, pens, or paper. they have to pass through a scanner to enter and leave the floor. They work on stations with locked down ports and restricted internet access.

I went there for a meeting but was not allowed on the floor because it was not pre-approved by security.

If the Apple centers where Siri errors are reviewed is like that, it would be hard to sneak recordings out.

This is actually how this whole thing started, a contractor handed a bunch of recordings to a news outlet: https://www.vrt.be/vrtnws/en/2019/07/10/google-employees-are...

So it's not about how long until it happens. It's already happening.

And what would you do with them if they leaked? It’s thousands of anonymized voices, basically meaningless unless it was some famous person and that person were identifiable. It’s like leaking a list of first names or a conversation recorded at some random bar that nobody has ever heard of with people nobody has met, nor cares about. Just noise. If they were tied to an identity, then it would be worrisome, but they aren’t.

I'm looking forward to it!

Look forward to that, something on my list of things to do. Though for the mic array, I want to try out contact microphones, make a whole house far-field array that way, might work, might not, but it will certainly be educational.

One aspect I've worked out is how to make a balanced contact mic using two piezo discs stuck back to back with a rigid layer in between acting as the pickup. Trick is that as the two discs are back to back, they effectively produce a balanced signal and that alone helps eliminate so much of the static/noise that contact mics due to impedance are susceptible too.

Doing offline voice recognition is pretty challenging on Raspberry Pi since it is relatively underpowered. Using pocketSphinx with RPi 3, it takes 2-3x realtime to do recognition, and the accuracy is underwhelming. Most DIY voice assistants handle voice recognition and speech synthesis with online services, defeating the privacy purpose.

If you would like to really take on Siri and Alexa, I would recommend using beefier hardware.

Just remember thousands of icloud nude leaked

Siri depends on internet connectivity.

The cost will be enormous.

But I expect the outrage will be very similar to the outrage we see now towards industrialism:

Yes, the costs are devastating, and a small minority perceive that devastation acutely (including me). But the vast majority of people think: oh well, would you rather be starving on a farm in the dustbowl?

So I don’t expect any kind of major political unrest regarding privacy ever. Privacy Within Society was a blip in history. After that it was either/or and privacy was a thing you had to construct through separation.

I see nothing wrong with smart voice assistants etc.

I only see a problem with it all controlled by few corporations and all the proprietary algorithms underneath closed up..

They know all about us and we almost nothing about them. .. or in a dystopian future, even "they" then do not know how all the proprietary libaries, training data and algorithms etc. of all the smart homes and self driving cars and autonomous killer robots anymore, and then some conflicting virus gets out of controll and ... Booom.

Someone needs to clarify what those costs actually are. Some people don't value privacy as much as others, and they aren't "wrong", they just have different values and priorities. Personally, I am quite fine having my smart voice assistant send my recordings to humans somewhere to evaluate to train the database. In fact I always assumed they were doing that.

For Apple, auto-photo identification happens on-device.