Hacker News Reader

Comment by falcolas | Hacker News Reader

> It’s not emitting “code”, it’s emitting individual tokens ... The “code” is the arrangement of those tokens, but that is determined by the weighting of the whole network

This theory of operation is not borne out in reality. It's been clearly displayed that these tools are emitting verbatim copies of existing code (and its comments) in their input.

It's even being seen in image generation, where NatGeo cover images are reproduced in their entirety or where stock photo watermarks are emitted on finished images.

And so, what can be traced back to individual sources? Quite a bit it would seem.

View on HN

Well yes, if you’re asked to memorize someone’s code, you’ll get surprisingly far too. The fact that models can do this isn’t evidence of anything. It’s a capability (or “tendency” if you overfit too much).

I think it’s pretty obvious that if you train a model to reproduce existing work in its entirety, it fails the “sufficiently transformative” test and thus loses legal protection.

But there’s nothing stopping you from re-coding existing implementations of GPL’ed code. I used to do it. And your new code has your own chosen license, even if your ideas came from someone else. Are you sure the same logic shouldn’t apply to models?

Are you sure you can do that? Because it is trivial to paraphrase code and text.

Sure. I’ve done it for Bel.

  (def no (x)
    (id x nil))

  (def atom (x)
    (no (id (type x) ‘pair)))

  (def some (x f)
    (if (no x) nil
        (f (car x)) x
        (some (cdr x))))

  (def all (x)
    (if (no x) t
        (f (car x)) (all (cdr x))
        nil)))

I don’t even have to pull up bel.bel to know that those are almost perfect replicas. I typed it on my iPad.

EDIT: as far as I can tell, the only diff is that all comes before some. https://sep.yimg.com/ty/cdn/paulgraham/bel.bel?t=1595850613&

I could have kept going through most of the implementation.

I'd wager that most folks would consider the code for some basic list operations to be a little different from outputting a complete specific implementation of a fast square root function with its exact constants, including the comments.

Why? They’re both basic functions. I’d argue that if pg had left comments and constants, I’d have replicated those too.

Perhaps my example isn’t as impressive as the model’s capability, but operationally it’s the same.

> But there’s nothing stopping you from re-coding existing implementations of GPL’ed code.

There is if you've read the code. You can still violate copyright if you're hand-copying an implementation of a feature.

Lawyers recommend "clean room" re-implementations when reproducing copyrighted functionality (when folks bother to ask) precisely to avoid the risk of code being copied verbatim.

Functionality can be copied. Code can not.

Even if that theory of operation were true, it seems like an implementation detail that would be largely irrelevant in a court of law. Infringement is infringement regardless of how complicated your method for committing it was. If your AI reproduces a copyrighted work verbatim without permission, surely you are committing infringement? If you would be infringing by doing it via copy and paste, I would think a court would be likely to find that you infringed when you used an AI to do it. It does not matter that the algorithm on the backend is fancy...

Personally, I agree with you. Others do not.

But irrespective of legality, my point is that attribution is not such a hard problem as people believe, because their mental model of what's going on is much more complicated than reality. I feel they are seeing it as a true AI (mostly non-deterministic and thus hard to observe) and not ML which is deterministic.

> It's been clearly displayed that these tools are emitting verbatim copies of existing code (and its comments) in their input.

Which makes sense when you consider that the sort of code that is getting reproduced verbatim is usually library functions which developers may copy and paste verbatim comments and all into their project, especially when you prompt the AI with the header of a function that has been copied and pasted often, so the weightings will in that instance be heavily skewed towards reproducing that function

So that should make it easy to attribute, yes?

Probably why, like the article says, they're planning to add that

> In an attempt to address the issues with open-source licensing, GitHub plans to introduce a new Copilot feature that will “provide a reference for suggestions that resemble public code on GitHub so that you can make a more informed decision about whether and how to use that code,” including “providing attribution where appropriate.” GitHub also has a configurable filter to block suggestions matching public code.

I think harder, as it is spammed around in all directions. It's easier to attribute a unique piece of code that appears in a single repo.

But boilerplate functions don't deserve copyright protection as they are not creative. Can I copyright print('hello world!') if I post it in my repo? Do I deserve a citation from now on?

For better or worse, AI is a combination of machine learning algorithms. And these algorithms are black boxes solely because we don't add observability to them - we aren't looking.

But there is a desire to understand why an AI provided the output it did (to increase trust in AI generated output), and so there's a lot of study and work going into adding that observability. Once that's in place, it becomes pretty straightforward to identify which inputs to a model provided what outputs.

I have never seen an ML researcher claim that understanding the effect of specific training inputs on outputs is straightforward given the size of these LLMs. Most view it as a very difficult if not impossible problem.

And yet it's a major part of the overall concept of being responsible with our use of AIs. Throwing our hands up in the air and prematurely declaring defeat is not an option long term.

It's a non-starter for no other reason than potential copyright infringement means the government becomes involved, and they will stomp on the AI mouse with the force of an elephant - the opinions of amateurs and the anti-copyright movement notwithstanding.

As such, AI Observability is a problem that's both under active research, and the basis for B2B companies.

https://censius.ai/wiki/ai-observability

https://towardsdatascience.com/what-is-ml-observability-29e8...

https://whylabs.ai/observability

https://arize.com/

Those companies are not solving the problem you are describing

"No they're not" and "no it's not" (simplified from the actual response) are conversation enders, so I'll follow the lead and let this conversation end.

Have a great new years!

Well it helps to be accurate in your statements

Observability is great but it doesn’t give granular enough insights into what is actually happening.

Given a black box you can do two things: watch the black box for a while to see what it does, or take it apart to see how it works.

Observability is the former. Useful in many cases, just not here.

If you want to know what LLMs are actually doing, you’ll need the latter. Looking at weight activations for example, although with billions of parameters that’s infeasible.

The output may be recognizable by humans as being “the same”, but an ML classifier can look at what is “obviously” a cat and classify it as a turtle. So while your statement is accurate, it doesn’t explain how “traceability” could really work—what is the actual mechanism of “tracing”?

There's a whole field of work around AI Observability - tracing how data moves through a model. It's intended to help identify biases in the input data, and to help answer the question of "why did it make this decision".

And if you can trace how data moves through the model, you can identify "that output came from these inputs" and add the associated metadata.

On your image generation point, something potentially interesting I saw on twitter/Reddit was about artists uploading “no to AI” images to artstation and then that logo almost appearing to vandalise AI generated art. I’m not sure if it’s a real thing or just people on twitter and stuff who don’t like AI art trolling though. Mostly because I’m not sure how up to date some of these models are being kept.

That's a joke/fake

> It's even being seen in image generation, where NatGeo cover images are reproduced in their entirety or where stock photo watermarks are emitted on finished images.

Can you cite sources? I’ve heard this claim repeatedly but have yet to see a good example.

https://youtu.be/kqPKNksl9hk?t=80 <- NatGeo cover

https://news.ycombinator.com/item?id=33061707 <- Watermarks (there's been a few linked here on HN)

The AI learns to generate watermark-like things, because those exist on a large fraction of its inputs.

It doesn't mean the rest of the output existed in the training set. It's perfectly capable of generating a completely novel picture, then slapping a watermark on it.

My point is that the watermark is deterministic. It is taken whole-cloth from an input, and reproduced as-is on an output. Thus, it is able to be attributed.

Stable diffusion is more like a paintshop artist who grabs bits and pieces of other art and melds them together, and less like a painter who creates from their imagination.

> It is taken whole-cloth from an input, and reproduced as-is on an output.

I respectfully disagree with this claim. It’s not as-is. It’s remarkably similar. That’s a big difference.

> Stable diffusion is more like a paintshop artist who grabs bits and pieces of other art and melds them together, and less like a painter who creates from their imagination.

I also disagree with this. The uncomfortable truth, imho, is that what Stable Diffusion does is FAR closer to what human artists do than we’d like to admit.

Human artists are perfectly capable of reproducing copyright infringing images. They just generally choose not to for legal/moral reasons.

> It is taken whole-cloth from an input, and reproduced as-is on an output

This is only possible for elements that are repeated hundreds or thousands of times. It's generally considered a failure of training data curation, if it's anything more than watermarks. The AI can learn how to reproduce the watermark, while being literally unable to reproduce the pictures it was on.

Neither of those are verbatim copies, do you have an example of a verbatim (i.e. exact, non-transformative) replica?

The quote does not make that assertion…

Do you have a source for this claim? This isn't really how generative models work

https://news.ycombinator.com/item?id=34187038 points to the two specific claims WRT stable diffusion. The co-pilot ones are easily found. Quake 3's "fast inverse square root" is the best known instance.

Reply on news.ycombinator.com