Hacker News Reader

Comment by gigel82 | Hacker News Reader

It's surprising how poorly labeled these images are; who is curating this collection?

Can't they crowd-source a proper labeling project - I wonder how much better things like Stable Diffusion would be if its training would include correct, complete labels for the images. I'm sure lots of folks would willingly spend a few minutes here and there to aid with the labeling if it means they get to enjoy the model for free.

View on HN

Those images are not really "labeled". They just scraped the alt text. A lot of the recent advances in AI have been by using lower quality large scale web data, instead of hand labeling. The noise will average out. Hand labeled data can be used for finetuning.

This is the same thinking that leads to incredible astrophotography using stacked long exposure images that appear to be pure noise individually.

Not really. There's sound theoretical basis for that approach in astrophotography. The sort of bias and or noise introduced by systemically poor labeling is much less reliably averaged out.

This is Laion-5B, you can read more about it here: https://laion.ai/blog/laion-5b/

Imagen and Stable-Diffusion both used subsets of this full 5.8B image set.

Is Imagen actually trained on a subset of Laion-5B and nothing else? I've heard they used huge internal data sets.

They have their own datasets and included Laion-400M, a subset of 5b that was released prior to 5b. You can see a short explanation in imagen's "Limitations and Societal Impact" section at: https://imagen.research.google/.

> While a subset of our training data was filtered to removed noise and undesirable content, such as pornographic imagery and toxic language, we also utilized LAION-400M dataset which is known to contain a wide range of inappropriate content including pornographic imagery, racist slurs, and harmful social stereotypes.

Could you imagine if the mediocre results we currently get from "AI" were mostly from poorly labeled data in every huge dataset and not lack of a scientific or technological breakthrough?, it wouldn't be the first time that too many academics are blinded by the wishful thinking that what they need is an eureka moment when what is needed is tons of dull repetitive work.

So for instance for photography generation what may be needed is huge amount of clean photos with obsessively detailed labels, maybe just the exact same single-point-lighting (the exact coordinates of the light being a data point, plus strength/lumens), with 8 pictures of each subject in black background: front, back, left, right, top, bottom, 3/4 mostly-front (AKA the corner), 3/4 mostly-back, and then the same 8 ones but with white background, then also include all the info possible: weight, height, width and depth, plus versions of the most common states of each object (ball: inflated or deflated; bird: flying, idle or walking), plus photos adding two subjects together (one dataset of woman with hat, another wearing the same clothes but without the hat, and one of just the hat without the woman), with properly labeled relationships so it's clear they all refer to the exact same hat (or lack of), you get the idea...

Then the most important thing for "image generation AI" may not be computing power but the most boredom-resiliant staff you can hire; of course that's just an example for photography, for things like text you may need an equivalent rigorous effort by a multitude of linguists.

I think for human generation you need to add a bone/skinmesh generator in 3d. So it needs to learn what the position is of the body in a 3d space and compare with the source images.

If you aren't Google, manually doing that with 5+ billion images might prove difficult, to put it mildly. Large-scale labeling is typically bootstrapped with smaller models and whatever manual data you have. What's being curated is the bootstrapping process.

Well, imagine a Wikipedia like org/effort.

That exists for anime style images in the form of Danbooru: https://www.gwern.net/Danbooru2021

Other mediums don't have enough people interested in curating and labelling 5 million images with very detailed tag lists for free.

https://commons.wikimedia.org/

So stable Diffusion has a img2prompt mode. I wonder if that can be used somehow. The prompts it has yielded for my personal images have been very descriptive and good. It would be interesting to see how different the img2prompt output is for the training images. I would love to measure it but I don't even know how to calculate this distance.

That would be circular, no new information would be provided to the model.

if there are any researchers reading this, I'd like to introduce you to booru porn galleries like rule34. millions of images meticulously labelled by dedicated enthusiasts

https://www.gwern.net/Danbooru2021

There are already trained models by the community on these datasets. It's called Waifu Diffusion [1] (not 100% if this is the most popular model).

[1] https://huggingface.co/hakurei/waifu-diffusion

The training used in the current round of image generators is Contrastive learning - the idea is that you don't use "properly curated" datasets - you just throw volume at the problem and let it sort itself out. Which makes sense - humans learn by the same mechanism.

The LAION-5B set in particular is filtered by it's CLIP embeddings - basically run a better AI to classify the images, and check that the two descriptions rate highly enough for similarity, which in turn lets the system have some confidence an image isn't completely wrong (also the step you filter out NSFW and other things if you want, generally).

You're right. I just searched for "milk" and was not expecting the NSFW results...

I wouldn't call this poorly labeled, 'milk' and 'breasts' actually are semantically very close.

"Can't they crowd-source a proper labeling project"

I'm very surprised how primitive it is. I put in a few names of uncommon technical objects and some of its images were close and others so far off as to be unrecognizable or totally unrelated/useless.

What it needs is a quick/instant feedback system that allows humans to rate an image against the query word. A rating scale of say 1 to 5 where 1 is an identical match, 2 a close match, 3 marginal, 4 ambiguous, 5 wrong/totally unrelated. If in place then likely thousands would make an effort to optimize the matching.

Google has been doing this for years with their various iterations of captchas.

I don't think that's feasible for anyone but someone like Google or another tech giant.

They could use Wikimedia Commons to get a smaller collection of better labeled images. Currently it's images from Common Crawl extracted by Laion, not sure if it already includes Wikimedia Commons.

part of our idea is that artists can help with labelling their own work

The same artists who have been pitching a fit about how the robots are coming for their jobs?

Pretty sure they are trying to figure out a way their art can’t be used to train an AI and won’t be willingly participating in making it easier.

You might be surprised. We have almost as many opt-in requests as opt-outs since we announced this today.

We don't see this as binary in the long term. Maybe artists want to release art from a prior period, for example, but withhold their current series until they move to the next.

I think we'll see people who want to post images of themselves in there as well. Like, being a 'playable character' you could prompt for to be in the artwork could be pretty fun.

Why is opt out a thing?

Who told you you could use the data in the first place?

It's copyrighted. Reproducing any of it is a crime.

Reading through your comments, it appears you're under the impression that we (Spawning) trained models using this data. That is not the case. We're building tools to help people to remove themselves from, or add themselves to, datasets used (by others) to train these models. We're hoping to make that as close to 0 effort as possible for all parties. We agree that artists should control how their works are used, and we're working hard to make that the norm.

> It's copyrighted. Reproducing any of it is a crime.

In which case it should be fairly simple to challenge it in court, no?

I’m skeptical that Al will replace artists. The existence of computer chess and go, though superior in every way, has not killed interest in human competition. I see no reason we couldn’t have the same thing happen with art: people see AI art as a novelty and something worth studying but otherwise preferring human art for its relevance to the human condition.

I agree. It was telling that in what I perceived to the non story of the midjourney created image winning the county art fair, many of the other prizes were for activities that machines have been better at than humans for decades. Fruit canning. Furniture construction. The replacement narrative is overblown, but these tools are going to augment human artists considerably.

The problem is that competition in chess, in games, has always been for the sake of the game itself and the fun in playing it. 'game playing' is its own reward, there is no other entity interested in paying for results of chess games, even 'interesting' chess games. If there was, you could setup instances of Stockfish to play against each other and dominate that market.

This is not true of the artist market. While yes, many artists get into it for the fun/satisfaction of producing their own artworks, the reason that they can remain in it long term is that there are other entities in paying for the results of their work. See: game studios, film studios, etc. Before AI, the only way to get the assets for these various projects produced was by paying artists/experts for their work. Now, the production of those building blocks can be automated.

Seems it's related to what alt tags the images are using.

labeling 6 billions pictures, even at 10 labels per second would take 20 years, so I guess nobody

Could be done by reCAPTCHA, although might be more difficult to classify the results (whether a user passed the test) compared to their usual challenges.

Not quite a CAPTCHA, but something similar has actually been done before.

The ESP game[1] paired two random people looking at the same image while a timer ticked down. The players had to enter labels that described the image, and if both players applied the same label, their score increased, and the label was associated more strongly with that image.

Well-known labels for an image were excluded after a while, so you had to guess less and less obvious labels in order to score as time went by. Once in a while the system would also assign test images with known labels to prevent cheating.

Apparently a lot of people played it because it was fun, so there wasn't even the need to pay them for labeling the dataset...

[1]: https://en.wikipedia.org/wiki/ESP_game

Reply on news.ycombinator.com