angusturner

Largely agree with this, although I think it would be interesting to formulate in terms of: "what is the mutual information between the fMRI scan and the stimulus".

i.e) is there actually more information than a few bits encoding a crude object category, which stable diffusion then hallucinates the rest (/ uses to regurgitate an over-fit image)?

Or are there many bits, corresponding spatially to different regions of the stimulus - allowing for some meaningful degree of generalization.

There is a strong, well-understood connection between deep-latent variable models (e.g. VAEs, diffusion models), and compression.

Many state-of-the-art compression algorithms are in fact based on generative models. But the thing is, the model weights themselves are not the compressed representation.

The trained model is the compression algorithm (or more technically, a component of it... as it needs to be combined with some kind of entropy coding).

You could use Stable Diffusion to compress and store the training data if you wanted, but nobody is doing that.

Oh my god, I'm glad someone else noticed this because it was driving me nuts. Deeply troubling that their case hinges on such a fundamental misunderstanding of the technology.

I bet a proper analysis of that toy experiment would conclude that none of the original data points are perfectly recovered: Only the underlying distribution / manifold is recovered, which really doesn't lend well to their argument at all.

Stable diffusion (or any likelihood-based generative model) is a learned compression algorithm. It is not the "container of compressed bytes". You can use a trained generative model to compress images, by combining it with some kind of entropy coding / arithmetic coding.

In this sense, stable diffusion is more analogous to the JPEG algorithm than it is to a specific collection of JPEG files. As it stands, the originals trainng data is not stored, even in a compressed way.

As someone working on singing synthesis, I know how hard it is to get that last 10% quality that makes a human listener instantly recognise if the voice is real or generated.

These are really impressive results! For anyone interested, my team’s singing work: https://youtu.be/LPy20zSWhZA)

I agree that adoption tends to be slow and there’s a lot of legacy systems that will need humans to look after them for a good while.

But I think you are selling GPT a bit short. Partly because transformers are incredibly general architectures, which make few assumptions about the domain. They are basically SOTA on every domain for which learning-based methods are applicable.

And secondly because ChatGPT can perform “in-context learning”. That is to say, purely by scaling next-token prediction, there is an emergent phenomena where GPT gains the ability to learn from the prompt it is given, and solve new tasks outside of the training distribution.

Hattie Zhou has a cool paper showing how you can teach GPT addition by prompting - a toy example, but it signifies something profound in my opinion. In the last 6-months I have shifted my own view from “scaling is a waste of compute” to a recognition that LLMs are a serious leap forward.

For something very hyped, I think it’s still under hyped and under appreciated.

If you haven’t seen it, I’d highly suggest looking at some of the recent research on in-context learning and reasoning in LLMs.

It is becoming increasingly clear that there is a phase-change in behaviour when these models get large enough, such that they can solve new tasks outside of the training distribution.

See work by Hattie Zhou or Laura Ruiz for example.

It is clear to those following the research or using these models that they are not just copy-pasting… (Even if you can cherry pick examples were the LLM recalls highly occurring dataset items like fast square root or whatever).

At this point I'm convinced we could have sentient humanoid robots living among us, and there would still be a contingent of internet users assuring us that "It's not really intelligent..." (presumably bc something about Searle's Chinese Room argument).

Reminds me of a joke about how the criterion for intelligence is simply whether or not it exists. That is, as soon we build it, it is deemed "not intelligent".

As the limitations of these models continues to recede, it feels like cold comfort to tell artists or programmers not to worry because GPT doesn't have "understanding" or whatever.

I mean, in a sense yes the training set is gospel. But these systems are also (generally) tested against held out data.

When you have to model 100TB of images with 4gb of weights there is no way this is possible without learning some kind of patterns and regularity that generalise outside the training set. Most generated items will be novel, and most training items will not be reproducible.

It doesn’t seem radical to suggest that the copying issue will continue to recede as we get better models.

And there are areas of research specifically concerned with _provably_ showing that you cannot identify what items a model was trained was.

Lots of reasons to be optimistic in my view.

I feel like a broken record on this topic lately, but I strongly believe that training ML models on copyrighted works should be legal.

It is clear to anyone that understands this tech that it is not simply "memorizing" or "copying" the training data, even if they can be coaxed into doing this for certain inputs (in the current iteration of the tools).

Ultimately, I think the problem of reproducing certain popular works or code snippets will be solved. One interesting direction here are the tools of information theory and differential privacy. e.g) proving that certain training inputs cannot be recovered from the weights, or that there is a threshold of how much information can be gleaned from a single input.

It is easy to imagine (because its nearly there already) a future version of StableDiffusion (or CoPilot) which provably compresses all training data beyond any possibility of recovery, and yet still produces extremely convincing results which disrupt the creative professions of art and programming.

Until we get to that point, it feels like the only consistent and sensible place to apply regulations is with the model end user. When I use CoPilot, I accept the small (honestly overblown) risk that _maybe_ I won't have the license to use some small snippet of code it spits out. But I'm happy to wear that responsibility, because the boost to productivity is so great that I dread a return to pre-CoPilot world. That is, a world where everyone keeps reinventing the same solution to simple problems over and over and over again.

the octopus would like a word with you

I think any argument against generative AI should not hedge on there being a fundamental difference in how humans and generative models work.

I mean sure, maybe humans aren't "just" doing sophisticated pattern matching, but there are good reasons to suspect this is some part of what we are doing. (Even if its not implemented with back-prop).

e.g) consider the work of people like Anil Seth, who propose that our brain is basically a generative model of the world, which aims to minimize the likelihood of perceptual data. (see also: Karl Friston's free energy principle). What's up for debate is how it is structured, what priors are built in, what is the learning algorithm etc.

Anyway, for all their limitations, it seems clear that current artificial generative models can: 1. learn hierarchies of abstractions, which 2. explain the observed data in the fewest possible number of bits, and 3. generate new, novel data based on the patterns that have been learned

If you want to describe this as "just sophisticated pattern matching".. then sure I guess? But I think there's a clear qualitative difference between this and searching for code in a discrete database (which imo would not be okay).

I get the impression that many peoples' grievance with generative AI (text, code, images etc.) isn't _really_ about the data provenance. Or at least, it feels secondary, compared to the general disruptive nature of the tech.

If tomorrow someone released a StableDiffusion, CoPilot etc with the same functionality, but respecting the provenance of the data (i.e. licensing etc), what concrete difference would this make? Programmers and other creative professionals would still (reasonably) be nervous about the implications for their livelihoods and communities.

At some point it will be possible to prompt a model for music in the style of <random artist>, and having never heard <random artist>, the model will generate a convincing emulation, based purely on statistical knowledge gleaned from millions of unrelated songs and text pairs. (I give it 5 years).

Now what? <random artist> should still be concerned (or not), but at least we're talking about the correct issue: How do we co-exist with generative models that massively disrupt/alter the process of doing creative or intellectual work?

Only tangentially related, but there is some cool work on using reinforcement learning to learn an optimal tax rate. See: “AI Economist”.

That just feels like such a pessimistic forecast to me. Of course, the current trajectory of improvements in model efficiency and better commercial GPUs / ML-accelerators may hit a wall.

But I would not be surprised if this was trainable on a commercial GPU at home within that time. But I think another important trend that we are seeing is that you don't need to train these models from scratch.

Open-source "foundation models" means that you can usually get away with the much easier task of fine-tuning, as to not throw away / re-learn everything that these large models have already fit.

Edit: I initially said 2-5 years, but on more reflection this does seem optimistic (for training from scratch).

A few months ago this task was virtually impossible. Then it was possible, but extremely expensive and pay-walled behind the "Open"AI' website.

As of about a week ago this tech runs on consumer GPUs. The weights have been downloaded 100s of thousands of times, and fine-tuning / modifying is possible.

Training from scratch is about $500k still, but it will only get cheaper and easier.

It seems pretty well-agreed that the HN moderation works because of dedicated human moderators and community guidelines etc.

I think spaces that effectively moderate AI art content will be successful (or not) based on these same factors.

It won't depend on some brittle technology for predicting if something is harmful or NSFW. (Which, incidentally, people will use to optimize/find NSFW content specifically, as they already do with Stable Diffusion).

Developing models that can predict if stuff is harmful ironically makes it easier for people to optimize for harm.

e.g. the one line of code in Stable Diffusion that predicts if stuff is NSFW, can be inverted to generate only NSFW stuff.

I tend to agree with OP that there is no technical solution to this problem.

Relevant earlier discussion about this issue: https://news.ycombinator.com/item?id=32436203

My bad, I misunderstood you! I agree with you.

Where I think this analogy breaks down is that these models are not simply copying / memorizing the training data. They are using the data to fit a model, highly compressing the specifics while holding on to general patterns and abstractions.

I really feel like "reading lots of books" is a better analogy. (Of course its not perfect, since these systems can scale way past the number of books a human can read).

These are fair points. I suppose I am more optimistic about all the potential ways this tech could augment or enhance human creativity.

Like, isn’t it kind of wild to think that the average person could have access to models that have learned from all art ever made…

I’m also encouraged by the fact that some of the best versions of these systems are open source and run on a commercial GPU.

I don’t think it makes sense to talk about these systems as though they have any agency (and I think I was careful not to).

Of course there is nothing that necessitates letting these systems do what humans do.

What I am saying is that I am allowed as a human to ingest copyrighted material, learn from it, form mental abstractions and use those learnings to generate new art. Why then, would I be forbidden from externalising some of that process into an artificial system that operates on similar principles.

The trained model obviously does not have agency. It is a creative and computational tool, much like any other digital tool or product.

Feel like people would have said the same with images 6 months ago tbh. (I mean sure, you could get a fake photograph of a person, but the text control that clip has unlocked is just wild…)

The question that fascinates me about this whole debate is on what basis (if any) one draws the line between a machine learning from copyrighted data vs a human learning from copyrighted data.

And I do think “learning” is the correct metaphor here. There are good reasons to believe that the brain is doing something conceptually similar to generative statistical models. i.e compressing inputs (/percepts) into some kind of abstract representation space. Even if we don’t understand the specific details yet.

As far as I’m concerned, it makes no sense to prevent training on copyright data. If I use a trained model to substantially reproduce parts of a copyrighted work, feels like the issue is largely with me and what I do with the content.

These models are just tools after all.

Good on them. As a researcher it is so disheartening when GPUs get hoovered up by people participating in proof of waste pyramid schemes.

Likewise, if google decides it wants to preserve its service for students and researchers, then I hope that people chewing up resources for malicious/questionable use-cases are booted off.

Isn't this just proof of work?

This is awesome. I use the "self control" app for Mac OS X, which is pretty effective, but obviously doesn't stop me using my phone.

Would be interested to learn more about this approach.

A lot of crypto trade (especially on shadier exchanges without KYC or the option to use fiat) is denominated in tether.

Tether keeps printing billions of dollars with no oversight or regulation. They lied about it being backed by cash reserves, and now make a much softer claim that its backed by "some cash" and other assets (other assets include more crypto IIRC).

Basically, it feels like tether is inflating the price of BTC, and the exchanges all have a vested interested in continuing to pretend tether is worth something. If there was a run on tether (i.e. everyone tries to exchange for fiat) it could be catastrophic for all of crypto.

(Disclaimer: I am not a financial expert, and have just picked this up from reading about tether over the last few months).

This is actually quite good for Bitcoin.