Hacker News Reader

Comment by aiiane | Hacker News Reader

Yes, if you choose to delete it, it will be deleted. (It takes nontrivial amounts of time to delete it across distributed service architecture but the deletions do get propagated.)

View on HN

What about old backups (e.g. tape archives, which I think I've read Google uses), how do those get deleted? Do you store and encrypt the data with a user-specific key and then delete that key upon receiving a deletion request? Perhaps user data is never stored as part of tape backups? Perhaps you have a system which (very slowly) is able to load the appropriate tapes and purge the user's data?

Cannot comment in too much detail as not sure how much information is public but tape backups are also in scope for wipeout and user data is removed from them.

Is any copy of data anonymized and still kept around in ml models etc.?

I'm not an expert in this area but from what I know, personalized ML models are rebuilt often from fresh data that is not anonymized so if you request your data to be removed, then it's gone from these models as well.

That said, there's also data without PII (anonymized) that is not attached to any specific user, that of course is not subject to this data wipeout as Google wouldn't know what to delete. Such data can be used for general ml models.

Not sure how it's done at Google but anonymization is tricky and it's easier to gather two forms of information from the start, one with PII and one without PII. I think most large companies are doing it like that.

EDIT: Just remembered one thing. At Google, even when building general models data that is rare cannot be used because it could be used to identify specific people. For example: If search query is used by less than X people on a given day then these queries cannot be used for model building.

Thanks for the detailed technical answer. It helps; much appreciated.

The standard way to delete things from cold storage is to encrypt each piece with keys that you can more easily to delete.

You can also, of course, read in the data, rewrite it with the deleted portion removed, and write it out again.

(Disclosure: I work at Google, though I don't know anything about how Google handles this)

Thanks for replying to so many questions. I hope you aren't too discouraged by the negative responses, your work is appreciated!

I also don't believe this. How do you know it will be hard deleted?

Edit: Nm, I see you're a Google engineer in this domain.

I guess for the writer of the blog it isn't something to point out, but it would have been nice if they explicitly said that.

Maybe it is just just the writer's POV, but I do worry they didn't explicitly say it is deleted as far as Google's storing of such data...

I simply do not believe this.

Unless they're audited by a source that can be trusted and have the findings made public, I will not believe it either.

You're welcome to believe whatever you want, but a decent chunk of my day job is making sure that it's true.

may i ask what your day job is?

Answered here while your comment was dead: https://news.ycombinator.com/item?id=19809515

'justcodebruh' your comment is marked as dead, but to answer your question I work on Google's privacy engineering team.

At this point the big companies probably need to submit to some external audits to really get trust back.

I've worked on wipeout compliance at Google and IMHO there's no possibility that an outside auditor would ever be able to understand those systems to such an extent that they would discover a flaw in them that wasn't previously known to the many engineers who work full-time on wipeout. The systems are just too complex. One does not simply `find -type f /google` to figure out what data is available.

Part of the point of the audit would be to demonstrate that you take it as seriously as you say you do. Not so much to check the fine details.

I don't personally put any value in those kinds of audits but Google has those. For example they have outside auditors for SOC and ISO 27018 and whatnot. Some dipshit from Ernst and Young swears that Google has such internal controls and processes.

In other words, it would be the same type of theater that most companies engage in. I know plenty of companies with "strict security standards" administered by the infrastructure department so they can pass an audit or some other type of security compliance. But most security breaches happen because of badly written software that the "compliance department" never reviews and they wouldn't know what to look for if they did.

Developers could have:

  sql = "Select * from Foo Where FirstName = '" + firstname + "'";

All over their code and no one in "compliance" would be any the wiser.

This isn't necessarily true. At my previous company we had a serious finding (DoD audit) as a result of poorly written software. The finding was discovered by running one of our standard internal reports and looking at who touched what, when, and where. Pretty straightforward, and although the fix was in code, the discovery didn't require any technical analysis.

Without going into confidential information, how could a report that didn’t analyze the code find possible sql injection attacks or over posting vulnerabilities?

Saying that systems are too complex to be audited is a horrifying statement that reveals an environment ripe for abuse.

That’s not what I meant. Outside auditors are the dumbest people on the planet and all they are prepared to understand is that you deployed Norton Deletemaster Enterprise Edition or whatever. Trying to explain to them that you’ve got a mathematical proof of why the backups are unreadable is a big waste of everybody’s time. I’ve been in that meeting and it’s worse than death. Still, the audits exist and the reports are public.

I disagree. Compliance enforcement of complex systems is not done by sending in a team of engineers to reverse engineer the whole thing. What you do is require the regulated corporation to maintain auditable evidence of its compliance in a specific format, then do routine auditing of the evidence. You only do a true deep dive into their systems if the routine audit shows non-compliance or potential fraud.

I value your contribution to this thread, can I ask some follow-ups?

Does Google have multiple "Deletion policies" such that deleting data from i.e. your GCP bucket follows one policy, and the "scrubbing" described in this article follows an entirely different policy? If so, do different deletion policies have different processes and different audit trails such that the end "deleted" state is subjective and controlled by the engineering and managerial oversight of the engineering/leadership team of that given product(s)?

From my (naive) opinion, it must be really, really, hard to for example, retrain every ML model that a now deleted datapoint ever touched. Its hard too to believe that, at some high level in Alphabet's org, there is no motivation to have the positive PR of feature(s) like this, but still at essence not delete the parts of the data trail that significantly drive Google's revenue. Do these datapoints significantly impact Google's revenue?

First let me say that I can't speak to whether anything "significantly impacts Google's revenue" - either I don't have any material information in that regard, or if I did the SEC probably wouldn't want me making statements about it. :)

So with that caveat in mind, let me see what I can help answer.

I'm not entirely sure what nuance you're implying when you say "different deletion policies" - while for instance Cloud might have a different timeline or set of triggers for when and what data is deleted, when it happens "deleted" still generally means "deleted". Some products like GSuite have the ability for administrators to say, disable accounts, which removes them from use but doesn't delete the account, but that's transparent to the domain administrator.

It's definitely nontrivial to track data propagation within large systems, but standardizing infrastructure, having central documentation of data handling plans, and having comprehensive privacy reviews for any new functionality that launches helps keep people on the same page.

Edit: oh, and regarding "retraining every model a data point touched" - the easiest way to do this is to just always be regenerating your models on a frequent basis. If you retrain your models once a day or once a week on a fresh snapshot of your data, they'll only ever be that stale.

Again, appreciate the reply. Myself and others post a lot of criticisms towards Google on HN, and though I feel much of it is deserved, its the commentary from actual Google employees that turns it into a much more substantial and informed conversation.

Uh, I guess all of this is wonderful and I appreciate the write up, but anecdotal evidence on an internet forum is not very reassuring to many of us. Until Google commits to some real actual transparency, there is no reason to believe they are actually deleting any data from their own servers, and that the options presented to end users is nothing more than a placebo switch.

No offense to you, but I remember when Amazon was releasing their home devices, and many people rang alarm bells in these forums only to be answered by supposed Amazon employees or friends thereof explaining why these devices couldn't possibly been sending data. Well low and behold, they are sending all sorts of data to Amazon. Were those commenters just lying? Were they misinformed? Were they trying to spread disinformation for whatever reason? Perhaps all three..

Here in America, the gig is up. Everyone, even our grandma's, understands that security and privacy is always going to take a back seat to profit. Always.

So again, no offense to you personally, but everything you are saying must be taken with a ginormous grain of salt.

The only answers are either open source or objective third party auditing, or preferably some combination of both. Words from google employees mean nothing.

What data are the Amazon devices sending other than recordings of commands which need to be interpreted?

Google retrains models all the time. They gave a presentation about ML and production last year:

https://www.usenix.org/conference/srecon18asia/presentation/...

You can see there's a section on privacy and deleted data as well.

Each team has its own policies, because each product is different: at a bare minimum they might be using different storage systems, but it's very likely that their data pipelines are quite different, too. In any case, each team's targets are at least as strict as any published ones, of course.

Also, it is likely related to the timeframes for auto deletion they are giving. 3 months is good enough for ML. If only a day, then it would not be worth anything to G.

And additionally, it's probably useful for external folks to know that each product team has a designated product counsel and [usually] designated/dedicated POC on several different security & privacy teams, all of whom provide strict guidance on what is or is not allowed, and why.

Yes. The former is governed by commercial terms, and the latter is governed by consumer terms.

That's different from your second question about model training/retraining -- there is an answer to that questions elsewhere (but it is taken seriously and training data is also deleted upon deletion of the source data where it contains PII). I don't know this next part, but I suspect models use such vast amounts of data that any arbitrarily small deletion wouldn't have much impact.

To be fair, companies do say things which they don't do or takes years to do it in the way that was advertised. Asking for a third-party auditor isn't unexpected or unwelcome in most cases.

I had a similar concern where I was unsure if Google deleted my data and then anonymized it for helping their models anyways. In my opinion, that would be materially different than deleting all of it.

I've beat up on Google a fair amount. But, companies can get in so much trouble for outright not telling the truth, I don't believe it would be worth it for them to do so. The only big tech company that has a history of outright lying and being provably shady is Facebook.

IBM. Microsoft. Tons of companies lie all the time.

Curious, as a Google employee would you believe a Facebook employee, on HN making the same claim?

No, because Facebook does not have a detailed written policy regarding user data deletion and retention, which Google does have.

> detailed written policy

Oh we're all good then!

I'm not sure if you're being sarcastic, but breaking a written policy is (AFAIK) fraud and has some teeth to it.

Assuming that you, an individual, can prove not only that the policy was broken, but also broken at the time when the policy was in effect. T&Cs change often, and only major changes require notifying the user.

The entire burden of proof would be on you, vs. a behemoth corp whose bottom line depends on maximizing data collection and retention.

Go to court on your own dime, prove it, and yes, the penalty might have some teeth. Might.

Usually AGs do that. They're slowly coming around to this being a major thing.

https://www.law.com/therecorder/2018/12/19/facebook-is-being...

The policy is up to 90 days to fully delete all user data, stated here https://www.facebook.com/help/359046244166395/

I believe you, but I also believe the NSA and others siphon that data and archive it so it's kind of a minor point if google keeps it or not, privacy is a relative term these days. We can be fairly confident since the Snowden revelations that data dragnet operations are continuing and expanding in scope.

Looking at all leaks coming from Google, including Dragonfly and Maven do you think that NSA could access data directly without any engineer noticing it and going to the media with it? I doubt it.

yes.. they split fiber pipes, if encrypted they use undisclosed zero days to steal encryption keys or try to break the encryption with super computers. these tactics are are all well known.

All Google internal traffic is encrypted, and I'd guess they use some sort of forward secrecy. I guess it's going to be basically impossible to use split fibers to sip at that data.

https://www.zdnet.com/article/prism-heres-how-the-nsa-wireta...

this is just naive

The change was made exactly in response to PRISM. In fact, it was something that was already rolling out at the time, but the disclosure of that program caused the effort to be dramatically accelerated. At this point, and for quite some time, they've been using encryption for almost all traffic on the internal network.

https://www.washingtonpost.com/business/technology/google-en...

you think that the thousands of NSA employees whose mission it is to collect and analyze that data just sat around and declared defeat after they started encrypting it? lol

I don't think the NSA has thousands of people to dedicate to the slurping of Google's data specifically. More like hundreds, if even that.

But I mean if your position is that the NSA has cracked modern encryption technologies, then I guess you better get off the internet. Whether you use Google or not, you're screwed.

i'm sure they've cracked some of them and can brute force others faster than anyone realizes. but they also are probably very good at targeting specific people/machines and getting the encryption keys they want, especially if its being used to encode a large amount of data. i keep a low profile and avoid doing illegal things so i'm not really concerned but it still sucks that privacy is disappearing because its convenient and profitable

That's not a very good argument.

Literally nothing whatsoever in Google's history suggests what you are describing is true.

Your comment here is the first direct indication I've had (at any level of trust) that Google does in fact delete things if you ask it to. Maybe it's dumb but I feel a tiny bit more inclined to trust these options now than I did before. Thanks for your contribution (to this thread and to deleting stuff).

Edit: Stray somewhat-related question just in case you'd know, why is it that if I open YouTube in a private browsing window on a computer with freshly cleared cache, it asks me which of my two Gmail accounts I want to log in with? Is it just IP address based plus some browser fingerprinting?

yes - and these online discussions are never astroturfed by employees of these companies.

Google's Sarbanes-Oxley compliance should verify that this works exactly as described. I don't expect those to be public.

You're welcome to all the tinfoil you can fit on your head but this is a written policy Google markets to their paying customers. What you are saying is they are just committing overt fraud.

https://cloud.google.com/security/deletion/

Fraud for which they get slapped on the fingers.

Brother it's 2019, Im not sure if you noticed but fraud is in big time. Markets are at all time highs. Fraud penalties are just over head.

Besides, for that document to work, the authors have to be trust worthy.. in this case the authors are not even close to trustworthy it might as well have been written by the hamburgler.

I'm an ex employee. There's actually a whole team whose sole job is making sure that all other teams have policies, measurements and alerting for deleting data. They'll chase you if any of the above doesn't hold. It's non-trivial work and slows down your development, if you believe in releasing early and fast. For everybody else, it makes total sense.

I bet it's not free to run, but it's cheaper and easier than elsewhere, because Google's infrastructure is built in-house and mostly integrated. I don't envy other companies that want to do the same.

> I simply do not believe this.

If you're not willing to put even that must trust in your various cloud providers, you aren't the target market. Surely you aren't sharing this information with Google already, right? So the feature isn't for you.

I've never seen audit reports from a trusted source of personal data being confirmed deleted for Apple, Amazon, etc. So I'm not sure what makes Google unique.

You don't have to believe it, but it helps rationally thinking about incentives.

What is worse:

- losing some data of the 0.1% of users who actually care, or

- the regulatory and PR nightmare that would happen if they don't and it is brought to light?

A subpoena for something someone deleted would be a good way to find out. My prediction is that for something deleted, say, six months ago, the subpoena would come back empty.

(Disclosure: I work for Google, though not on anything like this. Speaking only for myself, not the company.)

I work for Google, opinions are my own.

I understand your skepticism but at least in this case, I think your incentives are aligned with Google's. If we don't delete it (within ~30 days I think?) then I'm pretty sure we'd be in violation of GDPR.

Is that to say it's definitely deleted? I can't say that for sure since there bugs are always possible but at least I'm pretty confident the intention is to delete it.

So you delete the data from the training results of the services that ingested this meta? Yeh RIGHT!

Google may disassociate your data from your account, but your data lives on.

Reply on news.ycombinator.com