Laion Took Down Its Datasets After Discovery of Child Sexual Abuse Material | Hacker News Reader

Laion Took Down Its Datasets After Discovery of Child Sexual Abuse Material

404media.co

65 pointsorangeteacups89 comments

This dataset contains over 5.8 billion images, so 3226 of them being child pornography is a contamination rate of 0.55ppm. It's likely you will find this material anywhere there are large collections of images.

It’s 340TB, for reference. AIUI it was scraped from the web.

That seems less like an argument against removing/preventing the inclusion of CSAM and more like an admitting on behalf of those building these image sets that they're incredibly irresponsibly sourcing content for the aforementioned image sets.

I'm not saying this is easy to do, in fact I'd wager it's incredibly hard: but it seems increasingly obvious that there was no curation, no standards, no approvals process, nothing but a ramshackle mad dash to include as much visual content as possible to train these models on, with zero consideration of whether the people who created it were alright with their content being used that way, if the content was legal to use either by licensing or by being incredibly illegal just by it's own existence. We get more and more stories of the ethical lapses of the massive companies/organizations behind this tech and the resounding chorus from my fellow engineers is seemingly just "well just remove it and let's keep going," or shrugging shoulders "that'll happen, we're trying to do research here."

And fair enough but like, when you're doing research, you don't just run out and grab every chemical you can get your hands on and pour them all in a bucket and see what happens. How is this stuff not being found? What good are these datasets if they include this type of material? How can you trust the results you get from your training anymore once you know what went in?

Well I would consider a 0.55ppm contamination rate very low indeed? Mistakes happen in the creation of such datasets, it's human nature?

Just as we can't police a society to the extent it's completely free of crime, otherwise we will not function as a society and/or we will find ourselves living in a totalitarian state?

Including CSAM in your image set isn't excusable by "mistakes happen".

Also, the totalitarian analogy is total nonsense.

I think this kinda shows that you don't really understand the problem. If you had a human sitdown and review each image, there still would be images that get through. Even if you told him to err heavily on the false positive side.

Just to be clear, you are saying that no collection of images can ever contain any CSAM, ever [1]. If some activity might run the risk of including it, that activity must be stopped.

Is that correct?

If so, why are you not calling for the internet to be shut down? The collection under discussion is just a subset of the internet.

You'll also want to either ban or enforce review of all camera output, including closed-circuit cameras. Someone is going to have to watch all those watchers pretty closely...

As for totalitarianism, well, how are you going to make any of this happen? As we just saw you'll need client-side visibility into everyone's visual data stores. Not to mention old photo albums world-wide contain images that would set off ImageNet. There are famous paintings and statues that would, too.

[1] Do you exempt NCMEC? If so, how do you justify that?

You abstain from coffee, chocolate, and fish too I trust? Otherwise you're undoubtedly supporting trace amounts of slavery and child labor. I bet the device you're reading this with was also made with such (and if it doesn't then it's still supporting a genocidal regime).

Get real. Scraped data is going to have problems. The expectation should be that reasonable measures are taken to filter proactively and that reactive measures are taken to remove illicit content found after that.

They did curate the dataset – that's what the LAION-Aesthetics dataset is, which is what StableDifffusion used. I'm pretty sure they never recommended using LAION-5B directly as it has low quality images and would take more time to train and run. It's just a collection of links.

LAION is just a large scale scrape of the web, it's a bunch of images with the alt text. So you can say the same about the web itself.

> How can you trust the results you get from your training anymore once you know what went in?

Before taking a moral high ground, please take your time to learn how it works. LAION is well known for its notoriously poor quality simply due to being humongous. There's all sorts of unwanted crap in here. Everyone training on it knows that and does filter it, because it's practically unusable otherwise.

"Look everyone working in this field knows you can't just use the datasets as provided, you have to filter out the bad quality images, the dead animals, and the child pornography."

blank stare

Yes? If you crawl the web or use a result of that, you have to filter out the crap. That's... how it works?

Is really the idea that after taking an automated sampling of things posted online, some of those things will be things people post online, such alien one to you?

Somewhat reasonable position - except images were found that matched known and catalogued CSAM images. One would have thought that checking against that would be bare minimum for cleaning a large dataset?

Access to PhotoDNA and other 'perceptual hashing' APIs is not open or freely granted. It's not unreasonable that a team compiling a database would be unable to perform this level of cleaning.

PhotoDNA is explicitly free to LE, Nonprofits, and UGC-backed companies. LAION would have easily been able to get access to it for free.

In fact, at that scale, they would likely be able to get assistance. EDIT: Stanford themselves acknowledges in the original paper that bulk PhotoDNA access was given.

Did they even try?

Frequentist analysis is exactly the wrong approach here, for the same reason that one can't defend a crime by pointing to a much greater number of legal acts.

No, we accept road traffic accidents as fact and do not shut down entire road intersections due to deaths that occur. The same thing applies to child pornography, and 0.5ppm is certainly a tolerable amount in an extremely large dataset.

There are something like ~42000 traffic fatalities in the United States every year, we do not adopt draconian measures in dealing with it, even though loss of life and severe life-altering injuries occur. And such traffic accidents are absolutely devastating for the victims and their families.

Unless the dataset is particularly severely contaminated, the known illegal images should be removed and the rest of it should be allowed to stay up.

>No, we accept road traffic accidents as fact

Have you considered that maybe you shouldn't?

This is how nuclear was killed.

And deaths resulting from cancer and a whole list of other ailments. The act of living itself carries risk.

0.55ppm is low but you can still be prosecuted if that contamination is found on your drive.

The laws regarding child pornography, which were likely drafted by moral entrepreneurs[1] who created a moral panic out of it, are completely insane. Such persecution belongs in the 17th Century, not the modern age.

1. https://en.wikipedia.org/wiki/Moral_entrepreneur

The article notes that the images were parsed from the Commom Crawl dataset, so perhaps the outrage should be directed there? After all, it's a scrape of the Internet, and includes some pages with CSAM images on them...

How's this fundamentally different than Google (or any other search engine) indexing a site that contains CSAM? It's not technically feasible to filter every image on every indexed page during crawl time, especially if it requires uploading every image to PhotoDNA. The service would simply break under the load.

It's not different. If you tell Google there's illegal content in the index, they will remove it.

A lot of perl cluthiching in this article. Just remove the csam identified and reupload the dataset

Likely what will happen but because it's a strict liability crime it will warrant as complete a review of all data before re uploading to ensure there's nothing left

That's just completely ridiculous, I can't believe what the world's come to nowadays over such moral panics.

There's likely images of animal and human torture and mutilation in there. And there are people that fantasize about doing such things all the time. Nothing done about that, though.

I think, over a dataset of nearly 6 billion images, it's just impossible to ensure that it contains no child pornography.

Like trying to ensure all my kitchen surfaces are 100% free of germs. Not possible, or the amount of effort and disruption caused by the cleaning will make it impossible to get on with my life. Nasty stuff exists out there. We don't want too much of it, though. That's my opinion about it.

You have to ask yourself not “how impossible” it is , rather how much effort was put into mitigating this. The answer is probably a nominal amount but obviously not enough

I'll bet OpenAI and Google are also contaminated, but that we'll never know and it'll never be cleaned.

With open source efforts, we can eventually reach a state of public certification. It can be fully vetted, unlike closed source models.

The paper says that almost 500 of these were identified by MD5. That indicates that about 0 effort was put into mitigation.

So why all the worry about re-uploading a clean dataset because someone "could do a diff" to find the bad images? Could they not use the same md5 search to find the same?

i guess the argument is that these images were a needle in a haystack and now you have given the public a supermagnet to go find it.

like right now, if someone tries to download old laion and find those images manually, it would probably take them a very very long time. if they had the diff, they would find them in a few minutes.

i dont think its a good argument, but its an argument. the researchers found them without a diff so likely someone else can too. so theres not much reason to leave them in that i can tell.

Yes, I get that. But that's not what the researchers did. They used a database of know hashes. Can bad actors not do the same?

I don't think it's easy to get access to the PhotoDNA database. And it would be a service that people query against, so they'd have an audit trail.

You're not allowed to have the MD5 hashes these researchers used to find these images, so it's not possible to reproduce this result. These hash databases and APIs are tightly controlled, and only law enforcement, major tech companies and top researchers are allowed to use them.

How does that logic work? Unless your definition of "mitigation" is "modifying CSAM before including it in the dataset"?

Like, "identified by MD5" doesn't mean that they didn't attempt to remove images! A lot of CSAM out there is not yet on those lists, and a lot of it is being added as we speak, so if LAION-5B was filtered using one list of MD5 hashes, that doesn't mean it doesn't contain CSAM that's on a different list (or a newer version thereof).

And you have this problem no matter whether LAION actually took any steps to remove CSAM before publishing the dataset or not. Unfortunately, any sufficiently large such dataset will contain some bad content (how we should deal with this is a different question that I sadly don't have an answer for).

If it's all new, I guess that's one thing. But in general, if LAION wanted to know if this content was in the collection, they could have queried the same database. Some new heuristics turned up others, but I mean the basic search could have been done.

md5 hashes are not a reliable indication of anything, since producing images that have collisions can be done in realtime.

https://www.exploit-db.com/docs/english/46047-md5-collision-...

That's if you have control of both images. With a given MD5 hash, you can't make another file with that hash.

Still not an argument for ever using them as reliable evidence. You have to trust whoever made the original hash to not have generated it with a duplicate, innocuous image providing a collision. These hashes are not verifiable as they appear to be parts of secret databases, so you are trusting the authority of that database not to do something like make innocuous colliding images for the sake of violating your personal freedoms.

As the article points out, the problem is that by doing so you create a map for people to find CSAM just by diffing the datasets.

Good. Federal agencies can deal with that. Go shut down those websites and prosecute the maintainers. Seems like a freebie.

Get the dataset minus the illegal data back. It's essential to maintain parity with closed source efforts.

I'd bet that OpenAI also has this in their dataset too. And nobody will ever even know. Open source is the only AI that can be knowingly scrubbed and cleaned.

I think this is the correct response. Have federal agencies remove the content itself, instead of worrying about identifiable dead links.

These "illegal content" moral panics have ruined Internet freedom, they are now ruining AI as well. Such moral panics have been used as an excuse for absolutely heinous mass surveillance of the entire population.

The surveillance is for possible future crimes and is more debatable. We're talking about crimes we know have already happened. What would the argument even be for not addressing those.

Ruining AI? You don’t think what’s ruining AI is ingesting despicable content in it and then use this data set to train thousand of other tools with this despicable content? In what world don’t you think it’s not a problem?

It is a problem, just as crime in general in society. But our efforts to stamp it out completely could cause more harm than good. The price of liberty, is unfortunately crime. Otherwise if we go too far in preventing crime, we will be living in a totalitarian state.

if crime is done to me personally, it takes away my liberty. individuals have rights, which is the foundational concept of the Enlightenment and all modern society.

there is no panic here. every reasonable person agrees this is a problem, and it will take very little effort to fix it. these AI companies are not hurting for money. every single one of them is valued in the billions of dollars. they can spare "0.5 ppm" of their funding to go do what almost everyone agrees needs to be done.

I don't think that there actually is a low effort fix. Any sort of hashing or filtering strategy is inevitably going to miss some things. It's likely not possible to have humans manually review every image and even if you could, then you'd have the problem of traumatizing the people doing the review.

I'm not saying that more shouldn't be done, but that doesn't mean that the solution is easy or that a perfect solution even exists.

The primary obstacle to this is that a relatively easily identifiable diff would then be accessible for anyone motivated to identify the material.

How was the data collected in the first place? Why was the data considered useful for training if no one had evaluated it?

It's useful because it includes most of the web. In enormous datasets like this, introspection and filtering is done with statistical/ML techniques. This was done with both LAION (obviously not enough to filter CSAM out) and the models trained on it - statistical analysis, 3rd party model classification and captioning, aesthetic scores, perceptual hashes etc. Only the filtered part of the dataset makes it into the actual training set, as LAION has notoriously poor quality.

The point of various unsupervised and 'self-supervised' training methods is the ability to use unlabeled data, which enables you to use much more data than it's feasible for people to evaluate. Some crude automated filtering (e.g. deduplication) is still helpful, but removing the bottleneck of having a person look at every single item can enable you to use thousand or million times larger datasets, which is what's required to give models a 'general understanding' of a wide domain, by taking an (automated) look at literally everything that is available.

> LAION datasets (more than 5.85 billion entries) are sourced from the freely available Common Crawl web index and offer only links to content on the public web, with no images. We developed and published our own rigorous filters to detect and remove illegal content from LAION datasets before releasing them.

This makes me wonder if CommonCrawl WARC dumps themselves contain CSAM.

As noted in the quote, the CommonCrawl WARC files don’t contain images themselves, LAION used those files to find img tags and downloaded them themselves

Or -- someone just wants to take down the dataset so competitors could not use it.

Now that it's known to be contaminated with CSAM, using it is likely against US federal law. So they've now done their competitors a favor, and exposed themselves to liability. Seems unlikely.

What happened in response to the discovery that some blockchains have been contaminated with the same?

False analogy. The blockchain is not a tool intended for use in creating more images along the lines of the ones it contains.

However, any dataset of this size will also contain CSAM. Six billion images? Keeping it all out is completely impossible.

So this is effectively preventing the creation of any new models.

That's a good point... there's no (legal) way to prove the actual allegation here.

A question I haven't seen asked here (and maybe I am ignorant) but, dooesn't this kind of content exist exclusively in the dark/deep web? I thought CSAM in the clearweb was finally 1000% eliminated.

I don't think that's possible.

According to section 4 of the paper, over 700 of these were matches in the PhotoDNA database. I don't know if these have been perfectly scrubbed, but none of the popular image hosts would carry these.

No, there definitely is CSAM in the clearweb, it's just that it's usually ephemeral as it gets relatively rapidly removed as fast as it's added, and having a process to fight CSAM is (rather expensive) table stakes for anyone who wants to make a public service that includes user-generated content, because you will get "this kind of content" posted on your platform.

E.g. one of the issues with changes in Twitter moderation after the management change was that it turned out that reducing the moderation manpower meant that suddenly CSAM on Twitter was more prevalent.

The same applies to other media - e.g. observing Mastodon https://www.theverge.com/2023/7/24/23806093/mastodon-csam-st... found a pretty similar "CSAM rate" as in this Laion dataset.

Hm I guess that makes sense, if there is a CSAM "fill rate" in the clearweb of X images per unit time, and also assuming a "remove rate" that is approximately X then there will always be a "rolling buffer" of around X images constantly (which would change in its content every unit time, but always be there in terms of quantity) and that's probably what the algorithm picked up.

How do you know if pictures of genitals somebody uploaded are 17 year old or 18 year old genitals where there is no visible difference between legal and illegal content?

What about cases where one man’s cute family photos in the bathtub are another man’s pornography? There’s a bunch of reasons why that’s impossible.

I’ve heard a lot of the rational arguments as to why making digital images is nonsensical. Banning information is anathema to the nature of digital data just as much as it would be to make certain thoughts or ideas or shapes illegal, etc.

Are there statistics on child pornography over time? Is it spiralling out of control? Is it diminishing to zero? It would be easier to live with the pearl clutchers if there was lots of good evidence to show that owning illegal JPEGs causes either JPEG production or abuse offender rates to increase, or that decreasing JPEGs correlated with a downturn in offending.

It also might be ok to ban stuff because it feels like the right thing to do, but it’s not an amazingly solid argument.

I agree with your stance here so much. Possession data should not be illegal.

There are thousands of images of real murder[1] and other absolutely heinous acts committed against humans and animals out there. Millions and millions of murder fiction books purchased every year. So much violence in the media. And people with secret obsessive interests and fantasies of torture and homicide, who view large numbers of such images intentionally. And a large number of homicides each year.

We do not prosecute people for looking at images of murder on the Internet.

So comparatively little is done to stop that, instead we focus on sex-related moral panics, driven by what is likely a primitive primal human instinct.

1. War crimes, etc.

> sex-related moral panics

You’re not legitimately comparing the puritanical sex-averse aspects of the US and child porn, are you?

Yes, I am, when it comes to imprisoning people for simple possession of data, where there is absolutely no intent to distribute.

I think nobody should be going to prison for looking at or storing any image on their computer. Or reading any book or text as well (some countries prosecute possession of certain written material). Such actions are part of the private sphere, which the state has no business policing, unless you commit some violent crime or are severely insane, for example.

Sending people to prison for that absolutely is a moral panic in my opinion. It is a modern day form of heresy or witch hunt.

Murder for entertainment probably exists, but is vanishingly rare. Prosecutions of CSAM frequently involve people actively soliciting either the production of more or the opportunity to participate. CSAM enthusiasts network a lot. Read some court documents dealing with real cases rather than thinking purely in terms of abstractions.

Many of the prosecutions are for simple possession of such images. On top of that the "chilling effect" it causes means we can't download large collections of data such as historical web archives (e.g. GeoCities, etc), just in case there are illegal images in there.

The laws regarding child pornography possession are "strict liability" in many jurisdictions. So these laws prevent entirely legitimate activities from being carried out, because of the fear of being persecuted (it has parallels to the Salem Witch Trials).

> So these laws prevent entirely legitimate activities from being carried out, because of the fear of being persecuted

Can you name a single legitimate activity that is prevented from happening due to the need to comply with CSAM related laws?

Training a VAE on LAION-6B, for example.

Running open public WiFi at home. Because of fear that someone might download illegal pornography on it.

there is an enormous gulf, legally and morally, between fictional murder and real pictures of children being abused. which is the topic of the article at hand.

each person has the individual right to not have pictures of their abuse posted in public.

and as a society we dont need statistical evidence for every single law we come up with. some things we just know are harmful to victims because of common sense.

No, that's the same "common sense" we used to justify discrimination and other abuses towards people who were different in some way, e.g. by race, or by sexual preference (e.g. homosexuality). And at the time in history the discrimination was happening, most people considered the discrimination normal.

Heck even we still have problems with society discriminating against young people, and especially teenagers, by the way. So I really care about young people's rights.

But I still think these draconian content laws are unacceptable, they are replacing one form of abuse (pedophilia), with another by the state (sending people to prison for possession of data).

bigots will always try to abuse the legal system to attack minorities, that doesn't mean we shouldn't try to improve the legal system to protect victims of crime.

if we are going to argue over historical injustice, this crime in particular has been ignored for centuries because the perpetrators were often powerful and wealthy.

Regardless of if banning child porn is effective at stopping crimes against children (it doesn’t, porn in general correlates to a societal-wide decline in sex and sex crimes) that doesn’t mean all the CSAM victims out there want pictures of them naked as a child to be part of some uncompensated scheme to replace their jobs.

Absolutely, and we should do what's reasonably possible to reduce the number of images out there in circulation. But we can't do what is the equivalent of shutting down the whole Internet, in order to stop it.

Society and AI development have to keep going, otherwise progress will be disrupted. It's progress that has gotten us out of the barbaric Medieval period and the Dark Ages.

Good point. If someone publishes a photo then they violate the child’s right to privacy and I agree that they should be punished.

Doesn’t this mean that the crime is in the act of publishing? If they have the photo but don’t publish it then what have they done wrong? What about if the photo is a very realistic rendering of a child that doesn’t exist? (Or just a drawing, for which people have indeed been prosecuted in some parts of the world.)

(The flip side is already covered in law: if the face is real but the scene is fake then it is still an invasion of privacy.)

Here is the actual paper: https://stacks.stanford.edu/file/druid:kh752sm9123/ml_traini...

So they admit the dataset is containing external sources in a way that they are identifiable and countable (in general, not just CSAM). How is this legal copyright-wise?

Reply on news.ycombinator.com