Hacker News Reader

Comment by jrm4 | Hacker News Reader

So, and this is an ELI5 kind of question I suppose. There must be something going on like "processing a kazillion images" and I'm trying to wrap my head around how (or what part of) that work is "offloaded" to your home computer/graphics card? I just can't seem to make sense of how you can do it at home if you're not somehow in direct contact with "all the data?" e.g. must you be connected to the internet, or "stable-diffusions servers" for this to work?

View on HN

You can think of it more like this: If I do 100 experiments of dropping stones at variable heights and measuring the time it takes for the stone to land on the ground I have enough datapoints to make a linear estimation of gravity by using linear regression. So based on my data I create a model that the time it takes for a stone to fall is sqrt(2h/9.81). Now if you want to figure out how long it takes for your stones to fall, you don’t need to redo all the experiments and can instead rely on the parameters I give you (say 9.81 in this case) to calculate it yourself.

With these models it works exactly the same way. Someone dropped millions of rocks and created a formula of unbelievable complexity and what they now did is they released that formula with all their calculated parameters into the world. What you do when you ultimately use Stable Diffusion is you just calculate the result of this formula and that is your image. You never have to process those images.

This is exactly it. It’s pretty remarkable that it was trained on over 100 terabytes of images and yet the model has been distilled down to only 4gb.

Yes, and another reason for the small model size and the novelty of the underlying paper [1], is that the diffusion model is not acting on the pixel space but rather on a latent space. This means that this 'latent diffusion model' does not only learn the task at hand (image synthesis) but in parallel also a powerful lossy compression model via an outer auto encoder structure. Now, the number of weights (model size) can be reduced drastically as the inner neural network layers act on a lower dimensional latent space rather than a high dimensional pixel space. It's fascinating because it shows that deep learning at its core comes down to compression/decompression (encoding/decoding), with close relation to Shannon's Information Theory (e.g. source coding/channel coding/data processing inequality).

[1] https://arxiv.org/abs/2112.10752

Oh, wow. Now that you mention how it's similar to lossy (if not the same as) compression it all makes a LOT of sense. This is great. I teach IT and I already do a bit on how lossy compression works, (e.g. hey, if you see a blue pixel and then another slightly darker one next to it, what's the NEXT likely to be?) and this is something of an extension of that.

Correction: the auto encoder is pre-trained :)

Then maybe we should remind about this 25,000:1 ratio when an artist complains about his copyrights being abused. The model doesn't have space to actually copy his works inside, it can only memorise the equivalent of a thumbnail from each input. A very small thumbnail, scaled down 150:1 per width and height (square root of 25000). That's like a grain of rice on the screen.

That's not how it works though. Instead of applying arbitrary content detail reduction, the model is an attempt to distill the core of what makes a particular artist (or phrase, face, object etc) unique.

When programming, it will often take a long time and a lot of code to get to a few final lines that do what you want. You cannot say the final result is a "thumbnail" of all previous efforts. Rather, it is the apotheosis of it.

Some artists spend decades developing a style that looks like a kid could do it as well. Still, there is something unique in there, that a trained eye will recognize. Converting that particular style to a formula and making that freely available is at least somewhat morally ambiguous.

It's the same as someone trying to mimic a style. Nothing wrong with that. Certainly not something you could get copyrights from.

It is not the same as trying to mimic a style. It is cloning the essence of a style and making it readily available to anyone who asks for it.

Sure, it's not copyright infringement, but you could argue that this takes away from the hardship the original artist had to go through to perfect their style.

> Sure, it's not copyright infringement

Which was precisely what the discussion was about.

> But you could argue that this takes away from the hardship the original artist had to go through to perfect their style

You could argue the same things about photoshop, a lot of other digital tools, drum machines, the photograph and the phonograph.

Ah, I can step in here.

Fair use might work but maybe not? If I were to argue against it, I'd probably compare something like a recording of music vs. a MIDI file. Same raw data scaling.

That’s the interesting part: all the images generated are derived from a less than 4gb model (the trained weights of the neural network).

So in a way, hundreds of billions of possible images are all stored in the model (each a vector in multidimensional latent space) and turned into pixels on demand (drived by the language model that knows how to turn words into a vector in this space)

As it’s deterministic (given the exact same request parameters, random seed included, you get the exact same image) it’s a form of compression (or at least encoding decoding) too: I could send you the parameters for 1 million images that you would be able to recreate on your side, just as a relatively small text file.

So it's like a compiler which produces a 4GB executable file? And that 4GB is all the "logic" which can produce infinite possible images?

Not exactly. There’s no real logic per se, just data. It’s made up of tons of floating point numbers that define relationships to other floating point numbers.

> As it’s deterministic (given the exact same request parameters, random seed included, you get the exact same image) it’s a form of compression (or at least encoding decoding) too: I could send you the parameters for 1 million images that you would be able to recreate on your side, just as a relatively small text file.

For any input image? Or do you mean an image generated by the model?

I meant images generated by the model. Now that I think of it I could just send you the sampled vectors and you could feed that to the vector to image part.

My understanding is that images will not be bit identical due to GPU physics and decimal precision. Images from the same seed may be for all practical intents and purposes indistinguishable - but there are some flipped bits involved.

That's not my understanding. The same seed value to the device's random number generator should results in the exact same outputs - there's a bug being chased down in the MPS (MacOS) backend where the fixed random seed doesn't output the same image on different computers.

I've heard something a bit in between what you're both saying. For the same machine with the same seed / parameters [0], your output is deterministic. But once you change hardware or OS you will probably get bit-level differences that won't make a macro-level difference.

No idea how true that is, but on my windows machine, same params/seed is definitely deterministic.

[0] a help string in the SD source code recommends the ddim_eta parameter (which isn't exposed in most web UI or GUI's, including the OP github) stay at the default 0.8 for deterministic sampling. I have no idea if this means changing the value from 0.8 produces non-deterministic results with the same hardware/os/params/seed. Or if they just mean changing this from 0.8 will make your SD not match the online model but still be deterministic itself. But in my testing, changing this value gives no useful changes to the image generation, so I keep it at 0.8

If floats are used than there is no absolute deterministic behaviour accross different machines. It can never be guaranteed.

You could input an image and get it to recreate it as best as possible and then output a seed. That would be interesting!

This is a fascinating idea. Have StableDiffusion generate an image from the image you'd like to "compress" + a random seed. Feed that output to an adversarial network that compares source image to output and scores it. Try again with new seed.

After running for a while, the adversarial network outputs a seed, and you now have a few characters representing a reasonable approximation of your image.

I expect something after jpegXL will be a neural network based compression scheme, where the client has a n GB neural net attached. There have been several that already show promising results (it's likely to be more of a standards issue than a technical issue).

In 80s there was a man (forgot his name) who claimed that one day you could store an entire high res movie on a floppy disc. One day he might be right when AI can regenerate sequences of seeds to images/video. You just need a petabyte of models stored somewhere.

This is the main reason why attempts to say that these sorts of AI are just glorified lookup tables, or even that they are simply tools that mash together a kazillion images together are very misleading.

A kazillion images are used in training, but training consists of using those images to tune on the order of ~5 GB of weights and that is the entire size of the final model. Those images are never stored anywhere else and are discarded immediately after being used to tune the model. Those 5 GB generate all the images we see.

All those 'kazillion' images are processed into a single 'model'. Similar to how our brain cannot remember 100% of all our experiences, this model will not store precise copies of all images it is trained off of. However, it will understand concepts, such as what a unicorn looks like.

For StableDiffusion, the current model is ~4GB, which is downloaded the first time you run the model. These 4GB encode all the information that the model requires to derive your images.

SD has 860M weights for the main workhorse part. At 16-bit precision that is only 1.6 GB of data, which in some very real sense has condensed the world's total knowledge of art and photography and styles and objects.

It's not a search engine, it's self-contained and the closest analogy is that it's a very very knowledgable and skilled artist.

Is there a smaller version of the model available (<4gb) intended for use with 16 bit precision?

Diffusers shows how to use the fp16 variant.

https://github.com/huggingface/diffusers

What you interact with as the user is the model and its weights.

The model (presumably some kind of convolutional neural network) has many layers, every layer has some set of nodes, and every node has a weight, which is just some coefficient. The weights are 'learned' during the model training where the model takes in the data you mention and evaluates the output. This typically happens on a super beefy computer and can take a long time for a model like this. As images are evaluated the output gets better the weights get adjusted accordingly.

Now we as the user just need the model and the weights!

It’s all offline in 4gb file on your local computer. It’s like mini brain trained to do just one/few specific tasks. Just like your own brain doesn’t need Wi-Fi to connect to global memory storage of everything you experienced since birth, same way this 4gb file doesn’t need anything extra.

A kazillion images are used to create/optimize a neural network (basically). What you're working with is the result of that training. These are the "weights"

As someone with ~0 knowledge in this field, I think this has to do with a concept called "transfer learning" in which you once train with that kazillion of images, then use that same "coefficients" for further run of the NN.

Nah, transfer learning is when you take a trained model, and train it a little more to better fit your (potentially very different) problem domain. Such as training a cat/dog/etc recognition model on MRI scans.

The goal is usually to have the more fundamental parts of your model already working and you thus need way less domain specific data.

Here, you're not training anything, you're running the models (both the CLIP language model and the unet) in feedforward. That's just deploying your model, not transfer learning.

Reply on news.ycombinator.com