Hacker News Reader

Comment by FeepingCreature | Hacker News Reader

It kind of sounds like you're just giving a detailed technical explanation of how the network memorized some of the images.

View on HN

How so? I have an equation that makes a number:

  y = 2x + 3

And I have

  (x,y) = (1,5)

I'm saying the y = 2x + 3 is the ONLY thing stored in memory. The (1,5) is not stored anywhere.

(1,5) is training data. y=2x+3 is the model. This makes sense.

If the model captures (almost) all information in the training sample, that is memorization. In your example 2x + 3 memorized the point (1,5) because you can recall 5 as f(1).

This doesn't make sense. F(X) itself is not (1,5). You have to apply F for it to work.

Paint and paint brushes can be thought of as functions. You apply the right inputs (brush strokes) and you get an output that is a painting.

Therefore could you say that all paintings in the world are encoded into the paintbrush? No. You can't. Not until you use the paintbrush to copy something.

If I ask you to recite a poem you're a function f("recite poem") which returns the memorized poem. It's not even possible to have memorisation without a function.

But it's not memorized. The "poem" in the case of the machine learning model is constructed from a general algorithm. Similar to how 2x+1 has no concept of 3.

When you input a 1, it does a *2 operation and a +1 operation and produces a 3. Nothing is memorized. 3 arises as a side effect of unrelated transformation operations.

"*2" and "+1" is stored information that's used produce 5. The memory of a poem is stored information that's used to vibrate your vocal cords and produce a recited poem.

The memory of a poem is stored information but your vocal cord is NOT stored information.

"*2" and "+1" are vocal cords. They operate on any number of inputs just like how your vocal cord operates on any number of nerve signals and air blowing through it.

If my vocal coords are never used to record a copyrighted song then no violation of the law occured. If I never use the machine learning model to generate an existing picture then no law violation occured.

Well you can get around the whole memorisation problem by only using unique training samples in your dataset, same for Copilot. But yes, I do agree that in this case the user is doing the violation, not the network.

Unique training samples only makes the problem less frequent, it does not eliminate the problem.

The curve can still intersect a unique point.

It seemed to me all the examples in the article (and in copilot examples as well) are repeated training points, so I think it would solve the problem.

That just means only for those examples it was fixed.

Logically the mathematics behind it says that it can happen. You present evidence which is refutable by new evidence. Logic if correct, cannot be refuted.

The logic is also completely wrong, the dimensionality of a single image is very high, let's just say 2^(32*32) (if it was a 1-bit 32x32 pixel image). That's a probability of 5.56×10^-309 of converging to a sample by coincidence, which is to say it's completely impossible.

The logic is not wrong. First off your statistics showed that it CAN happen. Just with low probability.

Second your way of measuring the probability is incorrect. IF we were to randomly pick an image out of ALL possible images of 32x32 1bit images THEN your probability would apply but clearly machine learning doesn't do this.

The actual calculations behind the probability is actually highly, highly dependent on the dataset.

To take the 2D line analogy further if we perform linear regression on the points (0,0), (1,1), (2,2) (5,5),(329,329),.... and a bunch of points with the same x,y coordinates the line would touch ALL training set data points 100% of the time as the equation of the line would be y = x. This isn't even called overfitting, y = x is actually the ONLY solution available. There's no way to prevent this if the data is just really clean.

The set of natural images is going to be smaller than 5.56×10^-309, but it's still going to be astronomically large. You're simply not going to get an accidental hit on a 512x512 image, even with training. The fact that in the paper they only managed to reproduce photos from Stable Diffusion that are likely common in the training set proves this and conclusively disproves your hypothesis. In addition, they observe that unique photos are extractable from a larger model (Imagen), making it obvious that this reproduction is due to memorisation from the larger capacity of the model and is definitely not due to an accident (otherwise the probability to reproduce would not correlate with model capacity).

> The fact that in the paper they only managed to reproduce photos from Stable Diffusion that are likely common in the training set proves this and conclusively disproves your hypothesis.

There's no hypothesis. My conclusion is not scientific. My conclusion is derived from logic. It cannot be disproved.

>The set of natural images is going to be smaller than 5.56×10^-309, but it's still going to be astronomically large. You're simply not going to get an accidental hit on a 512x512 image, even with training

It might be low probability to get an exact pixel perfect match. But a similar match that is more or less indistinguishable (or even a different but obvious reproduction) to the original from the standpoint of the human eye is a significantly larger probability.

There's a joke programming language, HQ9, that only has three instructions, each expressed with a single character. One outputs "Hello world!", one prints the language's source code (quine), one prints "99 Bottles of Beer...".

Does the interpreter have 99 Bottles memorized? After all, it is a function (with a 3-element domain).

Reply on news.ycombinator.com