How we detect, remove and report child sexual abuse material | Hacker News Reader

How we detect, remove and report child sexual abuse material

blog.google

37 pointsmikece56 comments

Open articleView on HN

Completely missing from this blog post are:

1. Instructions on how to legitimately share photos with medical providers.

2. A transparent appeal process

3. An address for legal service.

The family that sent a photo of their toddler's penile lesion to their pediatrician is still left in the ditch.

Will their ever be an apology and reinstatement for this family?

While I wholeheartedly agree that Google should reverse the their actions in this case, I would argue that Google Drive/Photos is not the right "product" for medical (so, in the US, HIPPA-protected) data exchange.

The Google default is that your photos get automatically uploaded. And then it's Bambi meets Godzilla.

Google's AI would do better to refuse to upload and notify the user that the photo may be CSAM - AND give the user an opportunity to reply to a human AND explain the appeal process.

I feel like we'll never hear the full details of that story. It seems like a pretty easy thing to fix from Google's side. However, I think it's telling that he still remains banned and Google hasn't made any public statement. It feels like a constructed narrative piece. It's definitely a possibility that the father had CSAM in his possession and decided to go to the media, trying to get his account back, knowing full well that Google can't confirm or deny it publicly.

If they are manually verifying all of the hashes, and training ML to detect previously unreported images, that means they must be retaining a vast collection of material that they know is illegal. Not just passing it on with reports, but intentionally collecting it, retaining it for the long term, and processing it.

Anybody have any idea why they're allowed to do that?

They are allowed to do it because it benefits "law enforcement". See my other comment.

Most countries have legal clauses permitting storage for filter work.

Nothing in the article implies what you suggest.

There is plenty of legal porn imagery and legal non porn imagery and legal child imagery and legal adult imagery. It's not so hard to make your system detect if an image is similar to both porn and child.

Nothing in the article implies what you suggest.

On the other hand it seems one method is more likely than the other, given the legal implications of each.

You can dependably tag a 4 foot tall 18 year old as an adult?

> It's not so hard to make your system detect if an image is similar to both porn and child.

Yeah, I'm not so sure. Naked child bodies are very different than naked adult bodies, so it doesn't obviously follow that naked child bodies would be classified as porn without those being in the training set.

If the AI can "match" a picture of a black man to a gorilla, it can match a human child body to an adult body.

Flagging false positives for human review is within operating parameters.

That's pure conjecture.

When you put it that way, it is extremely disturbing to think that, statistically speaking, the largest consumer of child abuse material in the world might very well be Google (and/or Apple, for their version of the tech).

The people at Facebook/TikTok/<your favorite social media> responsible for filtering posts against policy that get mental traumata without being payed much are probably also serious contenders

Yeah, and I'd also throw a few three-letter-agencies' hats in the ring, like the NSA. Would be a serious competition.

But I think we all know who the true winner is: Vladimir Putin. His personal stash has petabytes of material.

Am I the only one completely bewildered by this comment? Putin seems like Not A Great Guy TM, but did I miss some expose?

Weird idea, but what would happen if you’d employ convicted pedos to review such content?

I guess you wouldn’t be able to fix them this way…

"pedo" is not a crime anyone can be convicted of.

I think the largest storage source should be the US government (and other world governments). This is as it should be since they prosecute cases and naturally collect evidence. Evidence can’t be destroyed just because someone was found guilty.

I can't speak to Google's implementation, but I highly suspect it is done in partnership with NCMEC. They are a non-profit that can work directly with private industry that was created by an act of Congress and can legally hold this type of material.

I’d be surprised if storing the hashes is illegal.

The hashes are fine. But to train image recognition models they’re going to need to retain the original images.

There's a regulatory framework and process for this sort of thing, you can probably find more details in the articles about Apple's recent CSAM kerfuffle.

https://en.wikipedia.org/wiki/National_Center_for_Missing_%2...

My understanding is there’s an industry standard corpus of images from child protection agencies used to train machine learning models.

This is just a great reminder that the cloud is just someone else computer!

Now tech companies are doing justice themselves.

I understand the good purpose for why they want to do that. Almost no one will contest the good side of that work.

But, think about it, if for the "greater good", they were to put camera+mic everywhere in your house and say that it is legitimate to monitor your home 24/24 for the occasional case you might be part of the minority of child abusers or, in the current trend, something similar like having hate speech at home and propagating fake news to your friends and family? This is how you finish with the 1984 world.

Even if being 'suspicious' of any individual by default without even a good cause, is like wiretapping the entire population by default in my opinion.

I don't envy whoever had to aggregate the training data.

"We also have a team of highly specialized and trained content reviewers and subject matter experts who help ensure that our technology delivers accurate results"

Wondering what does the training look like.

For your own sake, don't try to find out.

Must be literally the worst and most traumatic job I can think of in tech.

One of my friends at Google once mentioned it was an option for regular employees to participate. My friend also stated seeing Sergey Brin at one of the volunteer sessions.

Probably why he retired

> Our team members bring deep expertise to this work with backgrounds in law, child safety and advocacy, social work, and cyber investigations, among other disciplines. They are specially trained on both our policy scope and what legally constitutes child sexual abuse material. We regularly update this training and our guidelines in consultation with legal counsel, independent experts and medical professionals.

Sounds pretty thorough and covers all the bases that I'd expect for this sort of thing.

They mention suspending accounts responsible for uploading CSAM.

I'm curious what their process looks like when their own bots crawl a domain, find the CSAM, and then make it available to users via their image search.

What action is taken against domains hosting CSAM? How quickly do they remove the content from search results, etc?

Is it possible to make Google's filter to get false possible? I think if yes then it can be malused, similarly to incident with father and his son's medical image of groin.

Great!

If they decide someone has to go, all they have to do is say they found CP on their Google Drive. Who is going to believe the person when they say it was planted?

If you're having a hard time assessing the risk, imagine Trump was back in office and he was out to get all of the valiant anti-fascists.

Same with any other company that hosts files for you. Microsoft, GitHub, Dropbox, your email host, your server host, etc.

This doesn’t address the issue of banning and locking a parent out of a Google account from sending a photo to a pediatric medical provider during Covid telehealth: https://www.nytimes.com/2022/08/21/technology/google-surveil...

It's sad that this doesn't get the same attention as when Apple tried something similar.

Google's not using PhotoDNA (a preexisting hash-based list). They're using flawed image recognition algorithms to try to recognize never-seen-before CSAM. Apple's proposal was actually far better for privacy than this; and though it was done on-device, it had a threshold of 30 images. Google is doing far worse than Apple proposed, and nobody seems to care.

> This combination enables us to detect CSAM on our platforms at scale, while keeping our false positive rate extremely low.

Yeah, right!

https://www.theguardian.com/technology/2022/aug/22/google-cs...

People really need to understand that small percent times big number is a normal sized number. And you are pointing out one "false positive" that actually wasn't a false positive at all. You're not gonna be able to distinguish pictures of a naked child taken for acceptable reasons and not.

The story is Google not reinstating the account when the police said it was kosher which is a human problem.

They also conveniently forward to law enforcement parents who send non-sexual explicit images taken expressly for the purpose of examination by licensed medical personnel.

https://www.theguardian.com/technology/2022/aug/22/google-cs...

I've read a bunch of articles about this and I just can't see how there isn't more to the story.

Whether the original ban was automated or not, one of their specialists must have been involved by now, not least because of the media coverage and the fact they reported it to the police. While the police declined to investigate further (from what we know, last time I read about this), there must be another reason for the account ban to remain in place.

I'm biased, I work at Google (I don't know anything about this that hasn't been reported), but I also just don't believe that a company would wilfully ban accounts that they don't have a good reason to ban, after human review. It would be a stupid move, especially considering the media attention. Hence my guess that there must be more to this story. Again I have no insider knowledge, this is purely personal speculation.

Multiple independent organizations inside a company each with their own rigid and inflexible operating procedures. The user to moderator ratio in the tens of thousands to 1. How is this hard to see?

"I'm biased, I work at Google"

I don't know what you mean by the multiple organizations thing – this seems to be squarely in the realm of trust and safety. And while I understand the frustration in contacting a human sometimes, in this case it seems that a human will have reviewed the content and come to the conclusion based on their training, expert opinions, policies, etc (as per the article).

What can human contact add here? Excuses or reasoning for the content? Given the media attention, someone will almost certainly have reached out if there was any way that a reason or excuse could possibly change the outcome. If they didn't that implies again that there's more to this story.

I'm not going to pass any judgement on what the "more to the story" is here, I don't have all the facts, but it seems obvious to me that there are more facts, and that people are working from different amounts of information here.

The two pieces of information I think that stand out are: 1. Law enforcement elected not to pursue charges 2. The banned parties were comfortable enough going to a respected newspaper with the facts of their case, which, the newspaper presumably also did due diligence and sided with the users.

I don’t think that there’s really any “hidden” context here that would convince many (except maybe those who get their paychecks from big G) that google is not in the wrong.

Perhaps the hidden context here is that they also investigated the couple after the journal report and found other “violations” which were ban worthy after, not before, their case was aired publicly. I don’t think many people would consider this theoretical scenario fair either.

10ish years ago there was a big kerfuffle about a semi famous author who got his account closed, and then it turned out that he was semi famous for...writing fictional child porn. Some people argue they fictional / textual child porn should be legal, but Google would never allow that in their servers.

No idea about this case, but there are grey areas and the cowardly position is the legally/socially safest one, regardless of what one local police department decided. It doesn't make Google right, but it is what it is.

Policy #0: Never admit a mistake.

There are possible legal reasons they wouldn't reverse an account ban of this sort. There are also several instances of Google employees or their spouses being banned and AFAIK none have led to any criminal charges - e.g. https://twitter.com/miguelytob/status/1315749803041619981 (not sure if this one was for csam or not - but IIRC some were).

Police salivate over cases like this - until they decide to prosecute, it seems obvious that Google is in the wrong here.

No large business has ever said "I'll let my customers do whatever they want until the police forcibly stop them". It's too risky and unsavory. They set up a buffer around the law. Even legal adult porn is enough to get in trouble with Google.

> It would be a stupid move...

Please do remember that this company you work for is somewhat notorious on this website for making "stupid moves" seemingly every other day with respect to basic product management. The usual defense is that some internal incentive mechanism blindly optimized for these stupid moves and no one is at the wheel to change the company's high-level behavior... is there really any reason to believe things are different here? Is there a mechanism by which someone helping a user get around a CSAM filter would get them promoted at Google, or would they just be taking on potentially-arbitrary levels of internal risk going off the rails on a side project wherein they have to condone something that has an air of danger to it with the only real goal of helping a user who probably wasn't even paying Google anything in the first place? I admittedly don't work at Google, but the impression one gets on the outside looking in is that doing something that wouldn't cause some major impact on your performance review "isn't very Googley" :(.

And don’t restore accounts even when law enforcement decides not to pursue charges.

That is expecting Google to prove a negative about the user. They probably don't want to have the obligation to verify that the user really is innocent. And police inaction is not evidence of that.

What a weird non-article. No "how", just "yeah, we check for known bad files, and AI magic"

Reply on news.ycombinator.com