Don Knuth's MIP, 64 years later
nathanbrixius.wordpress.com
nathanbrixius.wordpress.com
Raymond Hettinger - Modern solvers: Problems well-defined are problems solved - PyCon 2019
Anyhow, in the late 90s I was given an SS10 (?) clone made by Axil, which was my main driver for a year or so, running RedHat for Sparc.
Why? This took me right out of the article. No one needs this energy.
It seems like OpenAI went out of their way to make everything generated look a certain way, while that’s clearly not a limitation of GANs in general.
It makes every single blog look like they have the same illustrator, which I really dislike.
So, it might be worse than a random stock photo in those aspects.
In this case, I'm glad that the image was included. I was about to close the tab but the image caught my attention, and I kept going.
"Immature poets imitate; mature poets steal; bad poets deface what they take, and good poets make it into something better, or at least something different."
The composer Igor Stravinsky "took" that and changed it to “A good composer does not imitate; he steals.” and after that quote was further changed in the LaTex documentation (believe it or not), Steve Jobs took it and changed it to be Pablo Picasso saying "Lesser artists copy, great artists steal".
It doesn't seem there's any evidence of Picasso saying it although Steve Jobs said a lot that Picasso said it.
It's just the '24 version of "You wouldn't download a car."
I think accusing AI generated work to theft, is a based on the wrong assumptions.
It's not just art. What do you do in computer science courses? Or math, language, physics, or any other subject? You study what has already been made. What you produce based on that knowledge is not considered theft.
Have you ever seen an AI image that, if a human made it, would be considered a theft?
You don't have to go to art school to produce art. Even if you do go, there's nothing stopping you from producing art unlike what you studied.
>We humans learn much in the same way as an AI. We study what has been made by others. I think accusing AI generated work to theft, is a based on the wrong assumptions.
What assumptions?
>It's not just art. What do you do in computer science courses? Or math, language, physics, or any other subject? You study what has already been made.
Most of "what has already been made" is not "copyrighted work for which you have no license or exception"
>What you produce based on that knowledge is not considered theft.
It certainly can be.
>Have you ever seen an AI image that, if a human made it, would be considered a theft?
I've certainly seen AI produce things that would be considered copyright violation or misappropriation of likeness or similar offences if they were to be used commercially.
Sure, but also you do, presumably, have to see to paint. The overwhelming majority of our training corpus is just visual observation.
AI can not.
Indeed, the problem is the utter unoriginality of the current wave of AI. Literally the only thing it can do is to produce the next token which is the most likely to be in the training set.
Could an agentic AI generate a cave painting if you restricted it to training data of animals and also limited its ability to painting with primitive tools? Sure, why not?
My argument is that art that diverges from reality is largely a function of path dependence and tooling more so than creative choice.
The historical context of the idea that “all art is stolen” is intentional hyperbole, it’s meant to be provocative and a little but humorous, and not meant to be taken absolutely literally like you are implicitly suggesting. It does not mean literally that thieves walked out with the paintings, nor does it mean that art progressed because people were hand-copying the Mona Lisa. It was referring to varying degrees of using other people’s art as inspiration, borrowing subject matter or composition, mimicking techniques, etc.
None of this is what AI does. AI is trained on fixations (to borrow the copyright term) and can only produce remixes of fixations. It can’t do the kind of stealing that T.S. Eliot was talking about. It doesn’t steal ideas, it only steals pixels.
The historical context is taking others' art into your own cognition and building upon it, adding your own spin. That is exactly what AI does.
The idea that AI can only produce remixes - putting aside the battles that were fought in the art world over the validity of remixes - is simply false. It was never true! It's an urban legend that's retold and retold because it'd be very convenient for certain arguments if it were true. But it just isn't.
I seriously recommend that you spend a week's worth of free time just playing around with SDXL, in a UI that's capable of inpainting. IMO, the quickest way to realize this idea is false is to actually engage with the system.
The authors of copyright laws, and the Berne Convention, did happen to write down what they intended, and they do make an actual distinction between stealing fixations and stealing ideas, and they clarified that stealing ideas is not covered by copyright laws. There are other IP laws that cover some forms of idea stealing.
> The idea that AI can only produce remixes […] is simply false
Why do you believe that? We know exactly how it’s built, we know what the algorithms do, and there’s nothing in there capable of original thought. It only interpolates, it does not extrapolate. AI is trained on examples, it builds a Markov model that reproduces what it learned, statistically, and locally reproduces what it was shown. We can prove it without engaging with the system, because we (humans) built the system. It’s deterministic and mechanical and repeatable. What exactly do you claim it’s doing beyond this? Be specific.
> Why do you believe that? We know exactly how it’s built
correct
> we know what the algorithms do
Dubiously correct, we know their function but not the purpose of the particular individual patterns learnt, see interpretability research
> and there’s nothing in there capable of original thought.
Incorrect, we have no idea what it takes to be capable of "original thought", the concept isn't even defined, we don't even know if humans do it. It's entirely possible that networks are capable of original thought, in fact there's a lot of evidence for it, like how if you put them in situations that aren't covered in the training they can often handle it fine.
> it builds a Markov model
Okay, do you know how the networks work? No, complete nonsense, Markov models have nothing to do with this. For one, LLMs are exactly not "locally" due to the long-range attention mechanism. Even things like RWKV, that "just" uses persistent state, can express correlations much, much longer than any Markov chain ever built. If you built a Markov chain for the sorts of correlation distances you'd get in LLMs you would need exponentially more states than there are atoms in the universe, which demonstrates that what's going on cannot be modelled as a Markov process without torturing the concept into incoherence.
> It's deterministic
Sure, it's deterministic plus a random noise factor. Anything in the world is deterministic plus a random noise factor, us included. If you perfectly recorded the thermal noise in a human brain you could deterministically repeat its output forever.
Imo this view is just "well the network can't be doing the same thing we're doing, it's too simple. I'm not that simple am I?" Yeah sorry, the human brain isn't that complicated.
> I don't think training is photocopying, and I don't think it's breaking copyright.
That’s a head in the sand position to take. There are thousands of examples, and dozens of lawsuits that demonstrate that literal copyright-breaking photocopying sometimes happens, I’ve seen it myself with Stable Diffusion and ChatGPT both. OpenAI is seeking copyright exemption for their models, which implicitly and openly admits they’re breaking current laws, and want the laws to change in their favor.
Even when output isn’t obviously breaking copyright, still, we know a-priori that we built a copy-and-remix machine by design. You can carry on arguing with everything I say, and that doesn’t change the fact that NNs are built with the intent to statistically reproduce their training data.
Side note that breaking copyright law does not require that the output is recognizable as a copy of one or more inputs. US Copyright Law specifies that one may not acquire and consume copyrighted material without permission. Some models are trained on legally acquired datasets, but many are not. Whether AI is stealing depends on the specific model and training data, so a blanket disagreement that no laws are broken is obviously incorrect.
> we have no idea what it takes to be capable of “original thought”, the concept isn’t even defined
May be true, but we do know for a fact that today’s neural network aren’t it, regardless of what original thought takes. Maybe we’ll crack the code in the future, and maybe today’s AI has started down the right track. Or, maybe not.
> plus a random noise factor. Anything in the world is deterministic plus a random noise factor, us included. If you perfectly recorded the thermal noise in a human brain you could deterministically repeat its output forever.
Maybe, but this hasn’t been proven or shown to be true, so is speculation. This armchair physics idea is hundreds of years old. Anyway, today’s AI is not deterministic plus a random noise factor, it’s just purely deterministic.
> Yeah sorry, the human brain isn’t that complicated.
Haha, the hubris. Prove it!
BTW this is straw man; I didn’t make any statements about the brain and I wasn’t comparing today’s AI to the human brain. Whether AI is stealing pixels does not depend on comparisons to humans or answering existential questions about consciousness.
Sure, when a picture appears in the corpus hundreds of times. I guarantee you a human artist could reproduce these pictures with the same level of fidelity, because they are cultural cliches. We choose not to, and when we don't, we do not violate copyright. So who commits the crime here? The model didn't decide to reproduce that image spontaneously, it was told to. The broad majority of its output cannot be shown to violate copyright, even with great effort. When a paper comes out with a model that violates copyright, always you see the prompt egging it on. That the model cannot tell that it is reproducing the input distribution a little too well, doesn't mean that's its default mode.
Besides, I suspect you don't want to rest your argument on a problem that can be fixed with the most basic effort put into dataset cleaning.
(Also, this only happened with some networks, as I remember the paper. I suspect it's a matter of training quality.)
So there are two questions: do companies violate copyright when training models, and do models violate copyright when generating images? I disagree with the former universally, and with the latter almost always.
> Even when output isn’t obviously breaking copyright, still, we know a-priori that we built a copy-and-remix machine by design.
Well, I just don't think this is true.
> statistically reproduce their training data.
Yes, on a high-dimensional abstract manifold, plus random noise. Again, at that point you've thrown enough divergences in that I genuinely don't see how humans aren't "mix-and-match machines" by this metric. Stop calling models names and make a concrete claim.
> we do know for a fact that today’s neural network aren’t it
Yes well I think they are it, or at least of the same kind. Show evidence.
> Maybe, but this hasn’t been proven or shown to be true, so is speculation.
Presumption of innocence still applies, even to code. I have my experience, you have your experience. In my time using models, I have used them to produce samples that were well outside the training distribution. They have managed this with no great difficulty. A mix-and-match machine would, by definition, not be capable of this, unless you use the concept so generically that it could apply to any computable function.
No you haven’t, because it’s not yet possible. Either you didn’t see the training data, or you don’t know how neural networks work, or you’re anthropomorphizing and drinking AI kool aid.
I mean, I'm just saying. I know what I saw. You're asserting I didn't see what I thought I did, because it's a problem for your argument. That's not very convincing though.
edit: Let me clarify, obviously the LLMs/diffusion models I've used can only apply the patterns that they have happened to have learnt. However, due to the high level of abstraction these systems are capable of, the concept of a "pattern" can be very abstract, and even include such things as "recognizing a pattern in the input" as an abstract pattern itself. As such, at the least the output is not constrained to some sort of trivial interpolation of the training set, but a deep nonlinear function of the training set, that has even partially fed back on itself. Again, arguably, just like us.
Which this one certainly was. Tape drives don't look like that nor does the arrangement make any sense.