Hacker News Reader

Comment by Flammy | Hacker News Reader

I'm curious if the police will push this, as it could reveal how much voice information is actually stored. Amazon has declined to answer how much voice information is stored online[1] and how much if that is tied to your identity. We do know that Alexa stores a list of queries you've spoken, and you can "delete" these records, but it is unknown if this impacts the actual voice data behind them.

[1] https://www.amazon.com/gp/help/customer/display.html?nodeId=... and https://www.washingtonpost.com/news/the-switch/wp/2014/11/11...

View on HN

Wait a minute, we'd already know if Alexa was recording everything by the network traffic that obviously has already been sniffed. We can assume that Amazon is storing every "Alex,..." query that you've ever done. So what is there to know?

We know they are streaming it to do all the NLP in the cloud. This is obvious from the computational costs alone of doing NLP.

The important question here is "What happens to the audio data streamed to the cloud, a few seconds AFTER the query has been identified (or not identified) and the results sent back to the Alexa device?"

The nieve assumption is "all audio data is deleted after processing". The reality is that data is still valuable to Amazon for a variety of uses such as

(1) further training their voice recognition software

(2) advertising data mining [how many people are in the room, things they are talking about --- note, Facebook's mobile app infamously does this]

If they just store the query text, that is 'best case' from a privacy perspective. If they just store query text and query audio, that is less than ideal, but not too bad. If they store all audio, ever recorded, for an indefinite period of time... that is what this police request could reveal: Audio data stored for a non-keyword-trigger, and for days/weeks after the fact.

No need for speculation: you can open the Alexa app and play back your own queries, audio and all. There's even a help section titled, "How do I delete Alexa voice recordings", which reads:

"You can delete specific voice recordings associated with your account by going to History in Settings in the Alexa App, drilling down for a specific entry, and then tapping the delete button. Or, you can delete all voice recordings associated with your account for each of your Alexa-enabled products, by selecting the applicable product at the Manage Your Content and Devices page at www.amazon.com/mycd or contacting customer service."

Once more though: that's just the audio flagged as a "query" by Amazon and stored as such. What about other stuff that gets uploaded prior to NLP & validation?

It's a serious question, and one that I think Amazon (and anyone else streaming audio to the cloud, Siri and Google Voice Search are only slightly less constrained) needs to have an answer for.

It's a great question. Reminds me of Uber and its new policy of monitoring location of customers few minutes after drop off "to improve service".

It is also true that if Alexa was listening in on conversations so as to have "social conversation context" for "improved processing of commands" ... well.

> What about other stuff that gets uploaded prior to NLP & validation?

I would be very surprised if Amazon sends everything up to the cloud; it would be very expensive to do so. Looking for "Alexa" in the audio is done locally, and probably triggers recording then.

I don't think the expensive part is accurate. Voice sounds okay using the G.729 VOIP codec at 8kbps. Capturing and storing that amount of data, for every Echo user, 24 hrs a day would be trivial for company the size of Amazon.

As others have pointed out, network monitoring has shown the Echo only transmits after it's heard the wake-word, so it would appear Amazon doesn't capture everything.

Hypothetically, if I was tasked with recording everything, I'd just add an internal buffer and then ship chunks of that data along with the regular queries.

Indeed, yes, that'd be the way to do it. You could claim the post-wake-word data is recorded at a higher bit rate than it actually is, thereby accounting for the larger than necessary data-transfer.

I won't be surprised if we find out this is happening. They'll probably call it a "bug", fix it in an OTA, but then accidentally regress 6 months later.

Something you should probably understand about Amazon is that their Leadership Principles are real, and customer obsession is their main focus.

A big part of customer obsession is not eroding the trust of customers by treating their privacy as extremely important. For example, I've never heard of an incident in the more than two decades that Amazon.com has been in business of them selling customer data to a 3rd party for marketing purposes or otherwise. They wouldn't do something that could erode customer trust, or would be anti-customer, because they are in business for the long haul. Customer trust can only be earned slowly, over time, but can evaporate instantly with one mistake.

I trust Amazon more than Google and others to protect my customer data, because they've frankly earned this trust over 20+ years.

If trust really is so important, then why are fraudulent and counterfeit goods so rampant on their store?

I won't be surprised if we find out this is happening. They'll probably call it a "bug", fix it in an OTA, but then accidentally regress 6 months later.

I'd be very surprised if this is happening -- Amazon knows people are watching, and it would erode trust with customers if it's found out that they are spying.

I'd be much more concerned about whether or not my phone is spying on me, even if it says it's only listening if I say "Ok Google" or "Siri". It's much harder for me to see what traffic my phone is sending across the cellular network than to snoop Amazon Echo's wifi traffic. And phone malware (sometimes baked in by the manufacturer) is increasingly common. Since the Echo does not allow user-installed Apps, at least it's less vulnerable to malware.

This is not happening. Period. Amazon takes customer trust very very very very seriously.

But you can't prove it's not happening. I'm personally skeptical that it is happening, but you saying "No for real guys, it isn't happening! I double-pinky swear" isn't actually proof of any kind either way.

Facebook, Microsoft and Google say the same thing, doesn't make it true.

I think you may be making a distinction where none exists. Can you give me an example of what "other stuff" you're referring to? (Disclosure: I worked at Amazon on a related product.)

If I have a conversation about coffee without once saying the word Alexa, will that conversation be stored by Amazon

As others are posting here, the Echo is obvious about when it's recording: when the onboard hardware detects the wake-word, the LED rings light up blue and it begins streaming audio to Amazon until the Alexa service determines that the query has been completed or that there's nothing more to hear. People have monitored the network activity of their Echoes to verify this (and if you don't trust that, well, nothing I say will help you.)

Sometimes the wake-word detection is mistakenly triggered and some unrelated audio gets streamed to Amazon, but the LEDs will still light up, and at times it'll even reply to a query that was never intended (I've been watching TV when my Echo will hear something and reply that I don't have any timers set, or whatever) but these recordings would still be covered by the link I pasted in the help text above.

I think a lot of the HN paranoia about these devices is overblown. All of this functionality is provided by devices that you already own, including hardware wake-word detection on all modern smartphones, which I assume everyone here carries around everywhere and recharges daily without much concern.

I think there's a stronger mental disconnect here in a lot of consumers. Detecting the wake word is easy to do locally; doing natural language processing on the resulting audio recording is what must be offloaded to Amazon, as it's too computationally heavy to perform that process locally.

Because the wake word is able to be processed offline, there is no need for the Echo device to record and transmit audio all the time. While the hardware is physically capable of doing this (and therefore, it will never be possible to fully eliminate doubt in the device's security) it would not be in Amazon's or the customer's best interest. Talk about a waste of resources, and a privacy nightmare!

Exactly. There is no cost model that accounts for the device constantly streaming realtime audio data to Amazon. Just doesn't make sense.

But it could still record background audio and transmit it later to Amazon, when an Alexa query is triggered. How do we know that it doesn't do that? Has anyone opened up the data stream to check what audio it contains?

You can measure the amount of data it's sending so you can at least verify it's not uploading vastly more audio than expected.

They could do sneaky stuff like upload the audio after the wake word + a few seconds BEFORE the wake word. The cost benefit of this doesn't seem high especially if you consider that Amazon says all the time that they want to 'earn customer trust'.

Maybe they have hundreds of secret wake words triggering recording and covert upload of audio. It's not hard to implement - they already have 3 wake words, and ability to record audio locally. A list of national security words and marketing words would be the first choices that pop into my mind.

Do you trust them not to waste this opportunity to get more user info? What about future software updates and especially user targeted software updates? They could avoid detection by only doing surveillance on selected targets. When they are caught, they can justify it with some "improving services" or "software bug" bullshit excuse.

> doing natural language processing on the resulting audio recording is what must be offloaded to Amazon, as it's too computationally heavy to perform that process locally.

This is an unquestioned assumption actually.

I think it's way more to do with the NLP system being very very proprietary.

I wouldn't be surprised if it could be done locally, computing is pretty powerful, but they never bothered to even try and make it work locally. Because "cloud computing" is a dream come true for businesses, hiding their special sauce, not having to explain what they're doing (whether it's hard or not) and always maintain control.

Typical audio compression rates can be 10:1, especially for voice. So if the LED rings are lit for 30 seconds, is there 30 seconds of network activity, or 3 seconds? If it's 30 seconds, either Amazon is grossly wasting internet bandwidth by streaming uncompressed audio, or it is sending 10x as much data as the query it is supposed to be sending.

You do understand that it is trivial to operate the LED lights and microphone independently, right? The hardware /could/ be designed so that the microphone can't be activated without also turning on the LEDs, but I'm guessing they're independently controlled by a microcontroller, and that it's the hidden software which is responsible for turning the LEDs on when the microphone is in use.

Yes, this is a concern for smartphones as well. The capability, even if it is not currently in use, is a concern.

I'm completely with you. Recording video from webcams without turning on the light indicator has been exploited for years. [1] The FBI director admitted to putting tape over his webcam. Why would anyone think audio is any different?

1: http://www.macrumors.com/2013/12/18/software-allows-hackers-...

And I'd like to add that even if Amazon doesn't do this, malware could, or certain state actors may compel Amazon to ship this on certain customer's devices.

Or if you have a conversation about coffee[1] including the word Alexa[2], but that cannot be parsed or validated as a "query" and produces no response from the device, does that audio get stored somewhere? Could it be captured in transit by other actors? Seems like an important distinction to me.

[1] Or, y'know, about some subject more subpoenable or privacy-sensitive.

[2] Or, y'know, not including the word Alexa but triggering the on-device recognition anyway. "Alex, uh, here's your payment for murder."

Sentences after the 'Alexa' key word are still recorded as un-identified questions that you can view in the Alexa app. Source: bought mom one, have used it.

According to others in the thread (i know nothing of this), it sounds like that is easily verifiable though, and not done. Ie, it sounds like they've monitored the outbound traffic of the device and it only sends Alexa... requests.

It's possible that they keep a rolling buffer of all audio in the device though. There could be legitimate reason to hold onto such a window of audio. Would make for an interesting discovery.

What would be the legitimate reason to do so? Queries are always preceded by the keyword, so what legitimate reason would there be to record and store audio before the keyword?

I could imagine trying to capture and analyze at least a few words leading up to the trigger word of 'Alexa' to help make it more accurate.

Even if the trigger word is Alexa, different languages, dialects, and slang might vary quite a bit. Some significant portion of people might use "Yo, Alexa...", vs "Hey Alexa...", vs. <a second or two pause> "Alexa...". I'm sure someone with a spouse or child named "Alex", "Alexander", "Alexandra" and so on might also trigger it by accident a bunch, e.g. "Alex, uh, who called?".

Matching all of those trigger patterns would help filter out false positive matches, or take false negatives and refine them so more people can use the product with their own language style and be less frustrated.

You can also configure the wake word as "Echo" or "Amazon" if Alexa is too ambiguous.

"What's the weather going to be like tomorrow, Alexa?"

Allowing that sort of sentence construct violates the privacy afforded by starting with the key phrase. If the Echo did allow this, I would not own one since it would send a lot more unintended sentence fragments to Amazon.

By saying "Alexa" first, it gives me a chance to hear the wake-up tone before I continue my sentence.

So if I'm saying to my friend Alex "Alex uuh, tomorrow we will rob the bank", I have a chance to stop before incriminating myself. But if I say "Tomorrow we'll rob the bank Alex uh... are you in?" and the Echo interprets that as a command, then it's too late, it's already been sent and analyzed.

It's strange that you'd have concerns about privacy when it comes to sentence structure and not that you have a literal black box with a powerful computer and an always on internet connected microphone in it.

Are you talking about my phone, my car, or my TV?

I trust Amazon more than Samsung, Chevy or, well, Samsung. But at least I can verify that my Echo is not sending home unusual amounts of data when it's not in use.

Isn't it really easy to run your own code on it? Not really a black box. Except well, literally.

I think they're talking about the distinction between the actual audio, which can be listened to and deleted from the Alexa app, and the processed data breaking it down into a (potential) query, possibly sitting nowhere but some query logs for underlying services.

The 'Alexa' voice recognition is done on the client, not on the server. Once the trigger word has been spoken, the audio data begins streaming to the server.

They carefully worded it so that while it is true it only starts transmitting when there is a query, it doesn't stop transmitting afterwards. It could continue recording and sending to the server for the rest of the day, not just the query (just kidding.. I hope).

> Facebook's mobile app infamously does this

Ummm source? I know there has been a bunch of paranoid conspiracy theories about this that are mostly backed by confirmation bias, but I don't believe anyone has actually came out with any proof.

Facebook as denied they are doing this, so I guess you're saying they're lying? http://www.theverge.com/2016/6/3/11854860/facebook-smartphon...

> computational costs alone of doing NLP

The speech recognition part could run on a Pentium 133 in the nineties (Nuance software, Dragon naturally speaking), the Echo device has probably a more powerful CPU than that.

The NLP part could run on the device as well, it's mainly a storage problem, you need dozens of GB, at least 32GB sdcard.

Ideally only the search phrases would be sent to Google/Amazon/etc. That's also how Bill Gates vision was described in his 1995 book The Road Ahead. A personal agent in a wallet PC would surf the information super highway for you. ...well we are "almost" there but with a different solution.

And that software in that era ha to be painstakibgly trained for each individual voice.

Echo "just works", and is clearly doing it by punting to the cloud.

Only the wake word needs to be processed locally.

Which the iPhone has no problem with. “Hey Siri” works even when you have no Internet connection.

Having the data locally necessary to process the wake word is slightly more practical to havig the language corpus necessary to process all English language in hundreds of accents.

Speech recognition in the 90s was absolutely turd.

Speech recognition worked fine in the late 90s. You could optionally train it for 15min with your own voive to improve the results. The Nuance software was better back then than what shipped with Win Vista inbuilt by default in 2006 and later. The same Nuance software runs nowadays on servers to power Siri, Google Now, Cortana. And who in the right mind would think every user gets its on dedicated super-computer. In reality, cloud based speech recognition and NLP has the advantage of a central database to collect different voice samples (to train) and a multi Gigabyte central NLP database, etc. But you won't get more CPU cycles than what would be available on a modern smartphone or a Smart TV or (what would fit in) an Echo like device. An offline-available software assistant as kind of premium feature for people who care.

So true - I vividly recall my attempt to dictate "Dear Sir" that ended up appearing on the screen as "Down Server" no joke.

I guess a way around this is:

1) Put the speech to text. 2) Put the text to speech. 3) Send the data over Tor.

Although #3 would use a Tor exit node and would allow packet sniffing/logging/mangling etc, it'd at least be more anonymous.

Thing is, this is in nobody's interest. Not in the police or Amazon's interest. For the owner, it'd depend on if they are the defendant/suspect or the victim. The victim is not always the person who owns the Amazon Echo. But an Amazon Echo is a nice point of defense because the proof cannot be deleted anymore. All one has to say during a robbery is "Alexa, call the cops". That's quicker than having to dial the number. And at any point, if you do say Alexa, you can also trigger a warning or proof. Ultimately, if you could change the keyword to something more generic you could trigger it without a robber noticing, or by tricking them. A keyword like "put your hands up"? I don't know.

I deleted my other reply to you. I understand the difference now. It's about the actual voice vs the query text.

"Alexa, remind me to murder John tomorrow evening."

> We can assume

That doesn't solve a murder.

"Hey Alexa, how do I dispose of a dead body".

Maybe the cops are just hoping to get lucky.

I guess Tina didn't like her roses.

but, are they storing every voice recording? How would you know?

So we know that they have a record of every query that Alexa has detected. I'm guessing what you're questioning is whether they're storing the actual voices so that they can differentiate between different people that have queried Alexa.

I'm quite confident that they store the voice data. It's very valuable to them as a source of training data for future versions of 'Alexa'-like products.

Definitely, as well as being able to eventually differentiate between mom, dad, and the kids.

you know because they let you access and replay the recordings from their app.

But we don't know that's every recording that they have. They very well could have a "allow_user_to_see_recording" toggle.

No, from day one people have been sniffing Echo traffic to see if it sends any non-triggered data. It does not.

I'm thinking more along the lines of;

"Hey we allow you to delete past recordings associated with your account and as far as you are concerned they will be, but we're gonna keep a copy for ourselves right over here, that you'll never know about."

Wow, so they set up the software to guarantee every user is running the same software? and that can't ever get updated? that's pretty amazing.

/sarcasm

Yes, they are storing it. Source: Amazon web site and Alexa app (which let you review / listen do / delete said audio). They are very up-front about this.

Reply on news.ycombinator.com