Ask HN: Has anyone leveraged GDPR to overturn automated bans? | Hacker News Reader

Ask HN: Has anyone leveraged GDPR to overturn automated bans?

146 pointsndmrs74 comments

There are many headlines of people getting their <FAANG/Big Company> accounts banned and losing access to a lot of important documents or services. Often, trying to talk to support on any of these companies is akin to talking to a wall.

However, GDPR has a clause stating that "The data subject shall have the right not to be subject to a decision based solely on automated processing". Which would mean that any EU/EEA citizen should have the right to have the decision reviewed by a human.

Has anyone successfully overturned a banned account using this method?

View on HN

From memory I believe FAANG etc all _claim_ that appeals you lodge are reviewed by a human.

Now if you don’t believe them then you’d need to take them to court and show why you think that’s not the case.

Which I guess means my question is why don’t you believe them and how likely is it that they are lying when they claim thy appeals are reviewed by a human?

There was a recent example with Google Drive where it explicitly disabled any way to appeal. I was able to reproduce the issue where it was flagging files that consisted of a single byte, sometimes followed by \r\n or \n.

Here's the HN story: https://news.ycombinator.com/item?id=30060405

Screenshots of trying to "appeal" (Request a review) from when I recreated the issue show pretty clearly there is no human involved: https://imgur.com/a/5YHQtLi

This wasn't an account ban, so I don't know how well it fits the GDPR language. Though I'd be surprised if this was somehow the only "fully automated account action" FAANG type companies are doing.

I don’t see how you get from Google’s statement “Was taken down for legal reasons and cannot be appealed“ to “no human was involved”.

I think the point is they, based on the file content, no human could have been involved in the decision. If there was a human involved, the files never would have been flagged.

The discussion isn’t about the initial flagging, it’s about the review.

That might be as simple as checking for the existence of legal documents claiming copyright infringement, or as reading a web page stating “we already removed X other copies of this file”.

Neither is a fail-safe way of doing such a review, but doing a thorough review might be expensive even for Google. Does anybody know how many such reviews they do each day?

It might also be a bug on their tooling to assist human reviewers.

Right, but I believe for those cases, the (automated) email also said that the ability to appeal was not available. So there is no human review, because there's no review at all.

Regardless -- and I know this is a "how the world should be, not how it is" type thing -- I really think the initial decision should not be allowed to be made by an algorithm. At the very most, an algorithm should be allowed to flag something for human review, but no action is taken until the human has a chance to review it and decide if the flag is warranted or not.

You're assuming the human both has agency, and gives a damn. It's more likely the human just rubber-stamps all bans, to get their KPI of number of appeals processed per day up!

If the human doesn't have agency, then it's not really a "human review", is it?

It still is.

Spirit of the law is a concept I would encourage anyone to think about when arguing about these things. I believe most people, and courts in particular, would not agree that a human rubber-stamping automated decision is in line with the spirit of the law. Clinging onto a technicality isn't going to go well.

I'd also like to point out that these laws don't just come out of nowhere in a vacuum, to be interpreted without any further context. In EU we have recitals and guidelines to give context and support the interpretation of regulations.

If you're interested, do read Guidelines on Automated individual decision-making and Profiling for the purposes of Regulation 2016/679 (wp251rev.01).

https://ec.europa.eu/newsroom/article29/items/612053/en

Here's what it says about human intervention: "Any review must be carried out by someone who has the appropriate authority and capability to change the decision. The reviewer should undertake a thorough assessment of all the relevant data, including any additional information provided by the data subject."

But the entire premise here is a "letter of the law" thing. Online account bans are pretty clearly not within the spirit of the GDPR restrictions on automated decisionmaking; note how the guidelines you linked, despite providing quite a bit of detail about different kinds of automated decisionmaking and rules around them, don't mention account bans at all.

There's only a handful of examples, and to me it is far from clear whether account bans would be in scope of the law. It's not meant to be an exhaustive list of all the things that are covered.

However, I could make the case that losing an account which holds years of your private correspondence and is your point of contact for private exchange, services you rely on (including where bills, account recovery emails, policy changes, warnings & alerts, 2fa codes, and other very important messages are sent), potential employers or clients, and which doubles as a login for other services (see openid) and so on, can have a significant effect on your life and could potentially fall under "decisions that deny someone an employment opportunity or put them at a serious disadvantage" or (admittedly vague) "lead to the exclusion or discrimination of individuals."

Some of the other examples in the guidelines seem mild by comparison (e.g. getting a reduced limit on credit card).

My perspective is colored by both having lost access to an email account and also being denied a credit card application; the former was a much bigger problem.

And you've just demonstrated why programmatic enforcement of contracts doesn't work. Courts are actually able to see through that semantic nonsense, because humans are able to discern intent.

Here's a human review process for you:

1) Check whether the output of the machine learning model outputs Yes or No.

2) If yes, ban.

3) If no, no ban.

This is not a human review process. The review process is algorithmic, the human is only involved to relay the result.

I think the chances that a judge would accept that argument is potentially a lot lower than you might expect.

it may still be human, but if it's just rubber stamping is it really a review?

They are required to have agency and give a damn. Of course, it is hard to (dis)prove that they actually do.

That's an automated process then.

Because once a human did get involved, via noise from here and twitter, Google admitted that there was a problem.

Also, just the absurdity that a human would review a file containing only "1" and decide the decision to flag it was correct.

https://twitter.com/googledrive/status/1486038872928792576

What's interesting is there's another user there that followed up 2 days after the tweet noting that other numbers weren't fixed. But this is ignored.

Problems shouldn't get fixed just because they got enough likes and reshares on Twitter.

Are you suggesting that Europe has established a fundamental human right to have Google provide free static hosting services?

No, they're suggesting that given that Google has chosen to provide free static hosting, Europe has decided they can't moderate it with purely automated systems with no appeals process.

This is like running a restaurant in the US, and not being able to discriminate by race. You're not required to run a restaurant, and certainly aren't required to run one that gives away free food, but if you are certain obligations come attached.

I'd also argue that your use of the phrase "fundamental human right" is misleading. Europe can and does require you do things for reasons other than respecting fundamental human rights. So does pretty much every other law making authority.

The EU has established that "The data subject shall have the right not to be subject to a decision based solely on automated processing, including profiling, which produces legal effects concerning him or her or similarly significantly affects him or her." (https://gdpr-info.eu/art-22-gdpr/)

Arguably the opposite where the EU may have in effect outlawed may free services be requiring human review of many activities.

I think reading the part where they say "I don't know how well it fits the GDPR language" would answer your question.

I think the better question is what a "human review" entails. I assume they have some kind of "human review" in there, but no meaningful human review.

> Which I guess means my question is why don’t you believe them and how likely is it that they are lying when they claim thy appeals are reviewed by a human?

Why would we believe them? It's Google's responsibility to prove their assertion, versus regulators taking them for their (not so good) word. The default should be the assumption that the corporation is being dishonest.

If you’re taking them (or anyone else) to court, isn’t the burden of proof on you?

Highly dependent on the law or regulation in question.

> Reviewed by a human

Can just mean some low-paid Amazon Mechanical Turk worker clicked on "Yes".

We seriously need an Internet Bill of [Personal] Rights and get it into law and use it against the FAANGs. Europe at least seems to be trying, along with California sometimes.

Agreed. We're overdue for a Magna Carta for our new era, lest we be absolutely ruled by the ever-growing myriad of algorithms and models that govern our participation in society and economy.

It upsets me that we've ceded control of a nontrivial part of our lives to a bunch of opaque risk scoring algorithms. No one knows how they work and even if they did they couldn't tell you because that could "help the bad guys". My mom just called me the other day and told me she got locked out of her Google account after trying to reset her password. Who knows what tripwire she accidentally walked into (maybe has to do with the fact that it's a big household that probably has 10 Google accounts on the same IP address). You can find it in a bunch of other places too. Discord will give you vague error messages when you try to sign up with a VOIP number that magically go away when you use a regular number. "Card processing error" when you use a prepaid card. Of course ReCaptcha won't even let you fill out a form if you have an IP address that has ever been associated with known undesirables.

For every proponent of a bill that protects user rights, there will be three opponents who claim censorship is needed to protect the public, to stop misformation, etc

Yes. I once got my account permanently locked at a well known service provider when I simply tried to make a payment for the first time. Support wasn't useful and all they could do was tell me that I somehow violated their Terms of Service for committing "fraudulent patterns" over and over again.

I could have and maybe should have just let it go, but it really got under my skin. I first tried out of band approaches to contacting somebody there. I didn't reach anybody, and you quickly realize how everybody else on the Internet just assumes you must either be lying or not telling the full story. Maybe it's just acceptable losses while doing business at scale.

So I finally just emailed them a polite GDPR request containing some spiel about Article 15(h), how I have the right to request my personal data, and also have the right to correct any inaccuracies in it, which must be the case since I committed no such fraudulent actions. I also requested a full list of all their data subprocessors, which I couldn't actually find listed anywhere on their site.

I'm not a lawyer, and I don't know if my request hit all the right notes or not. But literally one hour later, I got my account unlocked with a personal apology.

For what it's worth I also let them know that I'm not really looking to circumvent their systems, and I'm sure they have to deal with a lot of bad actors. But there really needs to be a better way to reach somebody to fix things when automated systems go wrong.

I also have the feeling that this approach would fall on deaf ears for big FAANGs, and there really needs to be some high profile ruling to put the fear in them.

> I didn't reach anybody, and you quickly realize how everybody else on the Internet just assumes you must either be lying or not telling the full story.

I have observed the same. When I evaluate service providers, I'm curious to know how they handle dispute with customers.. it's quite depressing to see that on most online forums, it usually goes straight into victim blaming. You must have violated the TOS, you must be doing something sketchy, you're not telling the whole story, you're just holding a grudge so get over it, you're just entitled, etcetra. There's very little sympathy, and no giving benefit of the doubt.

> But literally one hour later, I got my account unlocked with a personal apology.

Congrats! This is a lovely anecdote, thank you so much for sharing.

Having done a bunch of moderation work in various open source spaces, it's intensely aggravating that while the vast majority of complaints are from people who are lying about what happened there are also plenty of mistakes so stupid that they -sound- unbelievable and yet are in fact true.

And I'd note that I am very very certain that I've made mistakes that stupid over the years.

This makes me think of two things, but I can't quite put my finger on how they intersect, although they do feel extremely related somehow.

1) Hanlon's razor (do not attribute to malice that which can be explained by stupidity)

2) It was discovered (unfortunately missing citation atm :< ) that people who fall for spam scams that exploit gullibility do so "completely" - most see the scam for what it is, but the ones that do fall for it will kind of double down, defend their actions and see through their part of the "deal" because in their mind it Will Work. So it's almost like there's a super-thin line somewhere where everyone rubberbands to one or the other extreme ("are you serious, that's a scam" vs "are you serious, of course this is real") depending on whether That Last Single Piece Of Straw is on the haystack or not. (I just realize you could substitute "a scam" for "fake" and potentially explain a substantial percentage of conspiracy theorists... hmm)

The problem is that the GDPR is pretty much not enforced. See https://ruben.verborgh.org/facebook/ where the author tries to get all his data from Facebook - the case hasn't moved since 3 years now.

The regulators are useless (especially the Irish one which seems happy to shield big tech scum from having to comply with the law) which confirms my own experience raising complaints with the ICO (the UK privacy regulator).

Same goes the for 'cookie law'. A significant fraction of the web is in violation. The lack of enforcement sends the message that non-compliance is acceptable, so it's become the norm.

What cookie law? The one that states I have to make my website worse for everybody to use?

Yeah, I definitely ignore that law, and I wish 100% of website owners did. It feels to me like 99% of them follow it.

There is no law stating you have to make your website worse.

Making your website worse is just a what certain analytics providers want you to do so you keep paying for their services.

https://github.blog/2020-12-17-no-cookie-for-you/

I, personally, like it more when I can say "no, don't track me".

It's only worse for the user when the cookie notification is blocking the content, there is no "no, I don't agree" button or clicking it means clicking trough 100 extra toggles.

Set the do not track flag if you trust website owners to actually listen to your request. If you don’t trust them to listen to your request, then being forced to manually tell every website you visit not to track you is obviously pointless and worse.

It’s easier to trust a business to follow a law with teeth than to follow a mere non-binding header that politely requests the same thing.

What an utterly useless law. We have a convenient way for people to request universally that sites not track them. So let’s make a law that makes them have to ask “the right way” every. Single. Stinking. Site. On. The. Internet. Every. Single. Time. They. Visit. Every. Single. Site.

One might be forgiven for assuming that the law was actually intending to accomplish the reverse of the stated goal. It gives site owners tons of explicit opt-ins that nobody can complain about, even though they were coerced.

> We have a convenient way for people to request universally that sites not track them

DNT is utterly ignored to the point it’s officially deprecated in various browsers and the W3C working group for it disbanded:

https://en.wikipedia.org/wiki/Do_Not_Track

> It gives site owners tons of explicit opt-ins that nobody can complain about, even though they were coerced.

Coerced assent is expressly forbidden by GDPR, so we absolutely can and are complaining about this.

I believe GPC is considered legally binding under CCPA.

The DSA proposal also has language that appears to be intended to make such headers legally binding: "In order to avoid fatiguing recipients who refuse to consent, terminal equipment settings that signal an objection to processing of personal data should be respected."

> The problem is that the GDPR is pretty much not enforced.

17 companies were fined for GDPR violations just this month. Last year, Amazon was fined €746,000,000, Google €150,000,000, Facebook €60,000,000.

https://www.enforcementtracker.com/

I knew this link was going to come up so I've addressed it here: https://news.ycombinator.com/item?id=30141276

The 60M Facebook fine is a welcome development but my point still stands - how much did Facebook profit from breaching the regulation for the 4 years since it's been in effect? That fine should've had a few extra zeros at the end to actually serve its role, otherwise it's just a very small cost of doing business.

My understanding is that GDPR allows for increasing fines up to 4% of the revenue. But regulators don’t like going for maximum fines because there’s a higher chance the company fights back.

Regulators have brought quite a few successful GDPR fines. The reason that people think the GDPR isn't enforced, in my experience, is usually that they've been misled about what it does and doesn't require.

For example, the author you linked to is demanding a portable copy of all his personal data from all sources, which Facebook has no GDPR obligation to give him. He seems to have been misled by a form letter he found, which incorrectly conflates Article 15 data access (isn't required to be portable) and Article 20 data access (isn't required to include data that he didn't initially provide).

I don't disagree that there's been a few successful fines, but by that logic I should quit my job tomorrow because I happened to get lucky at the casino a couple times.

GDPR enforcement has been extremely lacking as demonstrated by the web being littered by non-compliant data processing consent forms. A compliant consent form should make the "decline" option as prominent as the "accept" one - the vast majority of services currently don't comply (including big names like Google or Facebook) and entire businesses such as TrustArc have been built on providing non-compliant consent forms as a service.

For GDPR enforcement to be considered serious, the fines amounts should be higher than the profits of companies built on abusing user data. If we look at https://www.enforcementtracker.com/?insights we can see that 1,6 billion euros has been handed out so far over a period of 4 years across the entire EU. How much does Google or Facebook profit in a year?

The entire experience of reporting violations is also a major problem and suggests the regulators (at least the UK one) aren't actually interested in enforcing the regulation. The process with the ICO requires that you first get in touch with the company and try to resolve your concern. This takes time & admin work on your behalf and a malicious actor can drag out the process for months. But let's assume that after you've done that and haven't gotten anywhere, escalating to the ICO merely results in them sending a letter. And when the company ignores that too, guess what happens? Another letter which they will promptly ignore too.

This sets the example that breaching the GDPR does pay, because not only reporting a violation requires so much commitment that the vast majority of people won't bother, but even once the violation is reported, the response from the ICO isn't actually an effective deterrent either.

> A compliant consent form should make the "decline" option as prominent as the "accept" one - the vast majority of services currently don't comply (including big names like Google or Facebook) and entire businesses such as TrustArc have been built on providing non-compliant consent forms as a service.

Nothing in the text of the GDPR, nor of any regulatory guidance that I've seen, suggests that the "decline" option has any particular UI requirements beyond merely being present. Again, while I don't want to claim that this or any other regulatory process is perfect, I think the primary reason people in privacy circles find it so frustrating is that they keep trying to enforce things that aren't actually GDPR requirements.

The following is from the ICO, which I don't think has any reason to interpret the guidelines any stronger than what they have to, since they aren't willing to enforce any of it anyway:

https://ico.org.uk/for-organisations/guide-to-data-protectio... :

> Consent means offering individuals real choice and control. Genuine consent should put individuals in charge, build trust and engagement, and enhance your reputation.

> Consent requires a positive opt-in. Don’t use pre-ticked boxes or any other method of default consent.

> Be specific and ‘granular’ so that you get separate consent for separate things. Vague or blanket consent is not enough.

> Make it easy for people to withdraw consent and tell them how.

https://ico.org.uk/for-organisations/guide-to-data-protectio... :

> What is an unambiguous indication (by statement or clear affirmative action)?

> It must be obvious that the individual has consented, and what they have consented to. This requires more than just a confirmation that they have read terms and conditions – there must be a clear signal that they agree. If there is any room for doubt, it is not valid consent. [emphasis mine]

At this point you could already argue that unless the decline option is as prominent (if not more) than the accept option then the user didn't actually intend to consent and just couldn't figure out how to decline.

> Consent should be given by a clear affirmative act [...] Silence, pre-ticked boxes or inactivity should not therefore constitute consent.

> The key point is that all consent must be opt-in consent, ie a positive action or indication – there is no such thing as ‘opt-out consent’. Failure to opt out is not consent as it does not involve a clear affirmative act. You may not rely on silence, inactivity, default settings, pre-ticked boxes or your general terms and conditions, or seek to take advantage of inertia, inattention or default bias in any other way. All of these methods also involve ambiguity – and for consent to be valid it must be both unambiguous and affirmative. It must be clear that the individual deliberately and actively chose to consent. [emphasis mine]

Seems like that's cut and clear.

What I would call "cut and clear" would be a specific description of how prominent the decline button must be. "Inertia, inattention, or default bias" is a very nonspecific phrase, and even if it does include UI, it's not obvious why the implied standard would be "as prominent" rather than, say, 50% as prominent.

Don't get me wrong, I'd really like companies to design this way on principles of general user friendliness, but I don't see much evidence that anyone involved in creating the GDPR intended to require it.

Not speaking for my employer, but the actual quote from GDPR is:

> The data subject shall have the right not to be subject to a decision based solely on automated processing, including profiling, which produces legal effects concerning him or her or similarly significantly affects him or her.

Emphasis mine. This would not include the vast majority of automated bans. It's more meant as a way to prevent e.g. automated police action via algorithmic selection.

I agree that it probably does not include the vast majority of automated bans.. but I'd prompt anyone interested to read the relevant guidelines to understand what might be in scope as far as legal effects or "significant effects" are concerned; it goes well beyond profiling by authorities, and commercial data controllers are far from exempt.

One example of a legal effect is cancellation of a contract. Examples of significant effect include automatic refusal of an online credit application, and e-recruiting practices without any human intervention.

Advertising is in scope too: "For example, someone known or likely to be in financial difficulties who is regularly targeted with high interest loans may sign up for these offers and potentially incur further debt."

Pricing is in scope too: "Automated decision-making that results in differential pricing based on personal data or personal characteristics could also have a significant effect if, for example, prohibitively high prices effectively bar someone from certain goods or services."

Finally, there's an example of profiling reducing a credit card limit. "This could mean that someone is deprived of opportunities based on the actions of others."

Anecdotally, getting kicked out of my email account has had far bigger effects on me than being rejected my credit card application.

https://ec.europa.eu/newsroom/article29/items/612053/en

This is correct. The Article 22 rights are the most narrowly-restricted of any data subject rights granted by GDPR. It only applies to things that are a Very Big Deal, like prison sentences, voting rights, or eligibility for government services.

While not tested by the courts, there is a plausible argument that "similarly significantly affects him or her" might apply to bans that impact your ability to earn a living. So streamers getting banned from YouTube, or AdWords bans for businesses where that's their main source of revenue. Bans that are lower-stakes than that get harder to justify under Article 22.

There’s also the right to be forgotten. How can they ban you if they have to delete all information that can be used to identify you?

Even hashes of your email address or payment data should be something you should be able to request they must delete.

Even under the GDPR, the right to be forgotten is not absolute or unconditional.

If a person revokes consent for their personal data to be used, the data must be deleted if "there is no other legal ground for the processing". But if a data processor has an overriding "legitimate interest" in storing data about you, then they have legal grounds to do so without your consent. The details of this will vary depending on the situation (and the jurisdiction) but, for example, fraud prevention is explicitly called out as a legitimate interest.

https://law.stackexchange.com/questions/37882/google-adwords...

Aside from using imap to backup my mail, what else should I do to help mitigate an arbitrary ban? I’ve had a gmail account for 20 years since 12 year old me got caught up in invite fomo. I’ve since moved to other providers but still there’s a fair amount tied into my account currently.

Mostly I’m scared of ‘multifactor’ where email access is considered a form of identity, but I’m not sure what else

getting anything sensitive data out of large companies with the GDPR seems to be impossible unless you want to resort to lawyers

I was trying to get my matchmaking data out of Activision Blizzard and they flat out refused, saying my data was their property

their exact response was:

> "the information requested are trade secret and/or intellectual property needed to preserve our game integrity"

I complained to the regulator, who agreed with my assessment, but to enforce it I'd have to go to court

seems the GDPR is basically useless

That's a very interesting situation for two reasons:

1. Arguably your matchmaking data is someone else's as well. Meaning, they'd be potentially exposing other people's data to you.

2. Arguably you don't own the matchmaking data. You only own the initial request for matchmaking. The end result is actually a product of their proprietary algorithm. You didn't generate it.

Perhaps it might be a good idea getting in to touch with a privacy campaigner, or if the European equivalent of ACLU exists, and have them test this in court because it affects two different and important aspects.

I hear people saying laws that require police reports, police enforcement or interactions with the court are useless. For people like yourself who feel this way, what alternatives do you propose?

what's the purpose of a regulator if they agree with you and can't do anything to enforce the law?

it's privacy theatre, nothing more

The purpose is so they can make rules about complex situations. It’s like how in the US the aviation authority (FAA) and crash investigator (NTSB) don’t really enforce the law, even if a criminal act contributed to the crash. They’ll either forward the information to law enforcement or leave it up to the insurance companies and civil courts to arrive at justice.

believe it or not not every country has the same regulatory system setup as the United States

(thank god)

the data protection regulators have no ability to create rules... their job is (supposedly) to enforce it

So you’re telling me ordinary cops and prosecutors in Europe are capable of investigating highly technical, civil crimes? I’m not an expert in European justice, but based on a few high-profile cases I’ve followed, this doesn’t seem to be the case.

GDPR Article 22 (the rule you refer to) also has exceptions:

> Paragraph 1 shall not apply if the decision...is necessary for...performance of, a contract between the data subject and a data controller

Which I can see applying as they probably have something in the ToS to enforce here.

It also allows automated decision making to comply with EU law. I don't know EU copyright law well enough, maybe Google has a responsibility to take down that data under copyright law and so this exception applies too.

The ToS does not constrain the company! The agreement does not stipulate that the platform _has_ to enforce ToS, so this is not a necessary action to perform the contract.

The data subject shall have the right not to be subject to a decision based solely on automated processing

Lots of leeway for FAANG/BigCo management to wriggle out of that one. "Sure, Jones in Legal gets an email notification every time an account is banned and has the option to review it."

I can only imagine the lobbying and "negotiation" that takes place to have legislators water down the requirement for real human beings to review or respond to such bans.

I doubt that would hold legally speaking because that would essentially be purely automated data processing.

Are these actual GDPR takedowns causing accounts to be banned, or are these via the internal copyright enforcement systems implemented so that actual legal GDPR requests don't have to be sent (with all the strings attached to those). ie. Banned because of GDPR (giving you rights) or banned because of violating Terms of Service (giving you no rights in almost all TOS)?

Reply on news.ycombinator.com