Hacker News Reader

Comment by satvikpendem | Hacker News Reader

If you've ever actually looked into the Laion datasets, you'll notice that they are hot garbage, in that the captions often don't even correlate to what the image is about, and the images are often low quality, bad cropped and so on.

There are other datasets being developed that use high quality images that are manually labeled by humans, such as by Unstable Diffusion who's having a Kickstarter right now [0]. They say they will be able to get a much more high quality model due to such high quality images and captioning, so we'll see. They also want to make the model and code entirely open source rather than the license that Stable Diffusion has which is not open source (it has many restrictions, enforceable or not, on the images made).

[0] https://www.kickstarter.com/projects/unstablediffusion/unsta...

View on HN

Looks like you missed the whole point of this dataset.

The idea that we proved is you can get a dataset with decent caption and images (that do match yes, you can see for yourself at https://rom1504.github.io/clip-retrieval/ ) that can be used to trained well performing models (eg openclip and stable diffusion) while using only automated filtering of a noisy source (common crawl)

We further proved that idea by using aesthetic prediction, nsfw and watermark tags to select the best pictures.

Is it possible to write caption manually? sure, but that doesn't scale much and won't make it possible to train general models.

> Is it possible to write caption manually? sure, but that doesn't scale much and won't make it possible to train general models.

Maybe, I don't think so however based on the above comments by Unstable Diffusion. It seems like people are underestimating the power of high quality data and just throwing the kitchen sink at models. Perhaps a set of good quality data can indeed outperform Laion-style datasets.

It's like the YC saying about doing things that don't scale, perhaps with the high quality dataset, we can train better models than CLIP and in turn use those to caption the rest of the images, only now the caption model is much better than previous ones.

I'd guess there is a bias-variance tradeoff. If you just want to make a certain kind of image, no doubt a manually labeled and curated dataset can be better. If you want a generic generative model that has learned a wide variety of stuff, scale wins.

I can see LAION playing a similar role to imagenet. The main application of imagenet isn't directly training image recognition models. It's pertaining on diverse data so that a "big" (big in 2016) model can be fine tuned easily on a small dataset, after learning to be a good feature extractor. From that perspective, the label quality (and concerns about bias and whatnot) are almost irrelevant

The new Unstable Diffusion model will be one of the several SD finetuned model out there, these models usually have much higher quality (but smaller image diversity) because they take the coherency of SD and costrain the distribution to a small high quality portion, this means that you can train a model on a smaller high quality dataset from scratch but you would not, for example, have the same level of coherency, this can only be obtained with an incredible amount of images, and they don't need to be "high quality", a man will almost always have 2 arms, 2 legs etc... regardless of the quality of the images, and after the model has fit the entire distribution you can finetune it to produce high quality and coherent images with a small dataset, that's why Unstable Diffusion will finetuned a SD checkpoint, also why researchers use these big dataset like LAION-400M/5B

> and they don't need to be "high quality", a man will almost always have 2 arms, 2 legs etc...

At the next generation it feels like the training set will be inbreeding on the flood of stable diffusion images with 7 mangled fingers, heads coming out of legs, etc.

LAION-400M/5B will obviously not change (and there is enough data to train a really good model), if a future dataset has AI-generated images, these will be highly curated as the images were chosen by the person who was using the model and probably further shared by other users, it would work like a complicated Reinforcement Learning from Human Feedback (RLHF), plus AI-generated images will usually have keywords such as "AI," "Midjourney" in the caption so that the model can learn to distinguish them from the rest of the dataset (and CFG comes to the rescue when the dataset is noisy).

It's possible to so a much better job with automation too. Context aware cropping, accurate aspect ratio, quality filtering by various metric... all solved problems long ago, but absent from Laion-5B for some reason. Perhaps it would be a good idea go collaborate more closely with image experts for the next round.

DALL-E, Stable Diffusion, GPT-3, Whisper, CLIP, etc are all trained on "hot garbage" and all of them are SOTA. Whisper is a great example, as it shows that this broader use of imperfect training data helps to make the models more robust and general than their "perfectly" trained counterparts. The trick behind all of these is to build mechanisms on smaller scale, human labelled data that can then be used to filter and label the broader dataset. Or use training methods that are more robust to imperfect data, like contrastive learning ala CLIP.

If you ever actually look into Unstable diffusion Kickstarter, you’ll notice that they’re not actually claiming they’ll manually label a dataset the size of Laion-5B - that’s a much bigger task than what you seem to think it is.

Even if a million people are labeling images, without any overlap, 5 billion images would mean each of them has to label 5000 images each.

What Unstable diffusion folks seem to be doing is that they’re using a few thousand labeled images to train a caption generation model and then use it to create a huge multimodal dataset with text and high quality images.

> If you ever actually look into Unstable diffusion Kickstarter, you’ll notice that they’re not actually claiming they’ll manually label a dataset the size of Laion-5B - that’s a much bigger task than what you seem to think it is.

I never claimed this either.

Creators of the data quality tool for computer vision, fastdup, continue to improve on their free release https://github.com/visual-layer/fastdup

Here's a short video of some recent results for LAION 400M https://www.youtube.com/watch?v=dlRCm29Upu4

Obviously there would be limits as to how much could be manually reviewed by hand (if 1000 people reviewed 1000 images each, only 0.02% of the images would be reviewed assuming no overlap was required), but I wonder if there would be any benefit to attempting to crowdsource captions for the dataset for the worst available images

It is not possible to manually label hundreds of millions of images to train a model on them, CFG exists to deal with this problem, also Unstable Diffusion will just finetune a Stable Diffusion model, so you cannot simply change the licence to what you want.

Boorus [0] contain millions of images, manually labeled to a pretty high quality. Notably defusion models trained on booru datasets have had good success.

This is not the only example of well curated image-tag pairs, especially in artistic circles. It's just that most of them are not CC.

[0]: https://en.wiktionary.org/wiki/booru

Booru use tags instead of captions, so a model trained on them is really limited; moreover, Danbooru has only 5 million images, while other booru such as gelbooru and sankaku have lower quality.

Tags are limited how exactly? Prompt crafting becomes a case of selecting the relevant tags, and the embedding space will still capture the dataset.

Danbooru is only one such example of well curated tagging and if we ignore copyright there are far more examples. These example just serve as evidence that refining poor labeling is not outside of the relm of possibility as you suggested.

A tag-based system would completely lack any kind of contextual information and it would not be possible to create any relationship between words; natural language is much more powerful.

An example, an image is tagged: kanna_kamui, kimono and torhu_(maiddragon), who has the kimono? Kanna, Torhu or both? It cannot be known, but with natural language it is possible to describe who is wearing what.

I think this is mainly theoretical at this point. In my experience, current technology doesn’t seem to be utilizing the additional information that comes from natural language all that well. For example: prompt Dalle2 for “a dinner plate on a stack of pancakes” and you will get ordinary images of pancakes on plates, not the other way around.

Edit: an experiment comparing tags/BOW vs natural language sequences in image generation tasks would be interesting to see.

I think this does not work mainly because of the unusual situation you describe, such as "a horse riding a person"; most of the time Dalle 2 is really good at following the prompt.

>It is not possible to manually label hundreds of millions of images to train a model on them

Citation, please?

I think you mean "the developers of this technology do not want to pay to have hundreds of millions of images labeled".

It is not believable that someone would pay humans to label 400mln or 5bln images/samples to train a model on them, but I guess if you argument is "everything is possible" then gotcha

You seem to be confusing "possibility" with your personal opinion on what you think would be done by others.

As a human being I know human limitations, explicitly labeling 400mln/5bln images for a particular task seems absurd to me, but if you think it is realistically possible perhaps you can give an example.

If it's done in a reCAPTCHA like way, it can be done fairly efficiently and for cheap. In fact Scale AI does just this, they do manual labor operations such as captioning images, as an API. Here's their product for image labeling: https://scale.com/rapid.

Unstable Diffusion is also doing their captioning like how I mentioned, with groups of volunteers as well as hired individuals.

Scale seems to do, for example, image classification but not captioning as it would be hard to compare the results with others people to verify the quality (when you have a discrete number of classes is really straightforward), also can you report where you read about the Unstable Diffusion plan for manually labeling image datasets? I want to dig deeper

From their Reddit post about this: https://old.reddit.com/r/StableDiffusion/comments/zhg18s/uns...

> We are releasing Unstable PhotoReal v0.5 trained on thousands of tirelessly hand-captioned images

They seem to have created a much smaller dataset than LAION's, it would not work to train a generative model on such a small amount of images (obviously the images here do not have a single domain).

The LAION dataset was designed for the broader community at the first place, so clearly the premise is that they don’t have millions to throw at the problem.

Reply on news.ycombinator.com