Hacker News Reader

Comment by wrs | Hacker News Reader

“…modify[ing] its AI model so that it traces attribution and gives credit to the original authors of the code, adding the associated copyright notices and license terms in the process…Biderman says is technologically feasible.”

Is it really feasible? What does “traces attribution” even mean here? It’s not emitting “code”, it’s emitting individual tokens that each were found throughout the input corpus. The “code” is the arrangement of those tokens, but that is determined by the weighting of the whole network, so what can be traced?

Can someone who understands generative ML better than me weigh in on this?

View on HN

Why wouldn’t it be feasible? (Maybe this depends on what you mean by ‘feasible’.) There’s no technical reason you can’t back-track the weights and make a list of which tokens from which training data were sampled. The list might be long, it could be impractical, but that has little bearing on whether it’s technically possible, right?

The problem here happens when the same source is sampled for many tokens in a row because it’s the only match for the context. It could also happen that many tokens in a row each have a long list of sources, but when put together have a subset of sources that appear in every token’s list. That means that someone’s input is being repeated verbatim, even if the network wasn’t trying to reproduce a single source. We could prune the list of attribution sources at the expense of compute by running largest common subset algorithms, which might be sufficient for attribution tracing?

It feels like this whole question might hinge on Fair Use. The network is copying other people’s code one token at a time. We (society & copyright law) all tend to agree that’s fine when it’s a single token out of context, and we all tend to agree it’s not fine when the whole output program matches any single input source. The question naturally becomes, “where’s the line, how many tokens in a row from a single source should be allowed?”

The token “if” appears in a heck of a lot of inputs, and a heck of a lot of outputs. What you’re describing seems like basically building an inverted index of the input corpus and then doing a search for the output. If the answer comes back with a high relevance then you consider it “traced”. I wonder what the size of that inverted index would be compared to the size of the generative model.

That sounds exactly right to me, or at least this is one specific way to implement a solution to the problem. Good question on size. Speculating… I would guess the size of the index relates not to the size of the model, but to the size of the training data. If you built an extremely naive and straightforward uncompressed index like this, you could imagine building the complete list of known tokens, and for each one the complete list of pairs of source index and character offset into each source. If a pair of 64 bit ints was sufficient for this, you’d have 128 bits per token for the index, compared to whatever the average size token is (which I’d guess is something like 2-4 bytes uncompressed). If that napkin math is anything close to reality, then I might expect the index to be ~8x the size of the training data (with error bars of like 2x-4x on both sides). Maybe we just call it an order of magnitude?

I think you'd find that most code matches (some) other existing code.

Some code, like the famous fast inverse square root, is widely shared. Even trivial code is often just copy-pasted from a popular SO answer.

Other code is driven by something like convergent evolution. It ends up similar to other code because of the limits of well known algorithms, language syntax, APIs, boilerplate, common coding styles, etc.

In other words, I doubt most human programmers would pass a "copyright check" against all existing open source code.

If true, how do you think these things should reflect on copyright or AI policies? Does it matter whether “most” human written code meets the same standard that we have for AI, and if so, why?

It’s hard to discuss unverifiable claims of “most” code. A lot of code, maybe most, or maybe not, is proprietary and kept within corporate walls, so we have no idea how much is or is not copied. SO answers are allowed to be copied, by definition. (And at least with SO answers, the code is publicly accessible, and attribution tracking after the fact is closer to possible, right?)

You do have a good point that attribution for some code could be practically impossible to track. This might get a lot worse if we allow AI to remix and republish it, that could even cause feedback loops if we’re not more careful about tracking attributions.

Copyright requires an element of creativity and originality. The easiest code snippets for AI to reproduce are things like language syntax (loops, definitions, etc), implementations of common algorithms, and boilerplate, and an index would probably find a lot of matches for those, but are they original or creative?

So the challenge for an index would be finding a rare case of creative, important code in a sea of trivial matches.

Incidentally, answers on SO are licensed cc-by-sa, so it's easy to violate copyright when copying them; but no one seems to care, suggesting that small code snippets are indeed "trivial".

Fair use is a bit more context dependent, especially when applied on other form of media. Unstable diffusion is a big example where many seems to feel that no fair use should be allowed regardless of how small of a token is taken by the AI. A small amount also depend on the original work and how it fit into society at large.

Having fair use to be the pillar that all AI training stand on is going to take a while.

Yeah good point, TBH I feel like the most practical way forward right now isn’t to wait for Fair Use to clarify AI, it’s probably going to take training data that’s better curated and partitioned into groups that explicitly allow certain kinds of ML use, and models that aren’t leaky across partitions (which sounds like what the problem is with Copilot in this case). This might mean new kinds of code licenses that specify whether it can be used for ML training or not, and companies are going to need to start respecting those licenses rather than assuming they can train on everything reachable from the internet.

Otherwise, if we are going to let companies ignore copyright and licenses because of their pinky promise that their models won’t be recognizable copies of any single input, then we really do need a legal criteria for how much of a single input is fair game, right?

The failure mode I see here, which seems quite likely, is that attribution would almost always list tens of thousands of source files, or more. They doesn’t seem particularly useful or meaningful.

Yeah, correct, I would expect the same thing to happen if you list all attributions. That’s why I mentioned a criteria for filtering by number of tokens in a row for a given attribution, and one possible method for doing the filtering via largest common subsets.

While the very long list would be impractical (but not infeasible), sorting and filtering the list by how much each source contributes would make it easier to manage. Pruning all the entries that are below a contribution threshold would almost certainly shrink it to a very small fraction of the total number of attributions. We don’t need to list all the attributions that contribute to only one token, all we need to know is which attributions ended up with 5 or 50 or 500 tokens in a row, right? Likely not many.

The judge/law will say it comes down to intent.

Here's my take from an audio DSP viewpoint.

A composer listens to 10,000 hours of music. One day she writes an original piece based on the annealed parameters (temporal, spectral, intensity, pitch, sequence...) of a million artists. However it sounds like another specific artist who sues.

It is not a cover, a remix, homage, or even forged in the genre... it's just accidentally a bit too like it. (compare: Banana Splits vs. Bob Marley - and - Huey Lewis vs. Ray Parker Jr. - which completely misses the real impact of "Pop Muzik" by M)

The question is, was she exposed to influencing materials incorporated without intent or was it plagiarism (intent to reproduce a derivative etc) ?

By contrast I can take a piece of music, break it down by analysis into melodies, chords, timings, and use FFT to extract the precise spectrums of instruments, feed those to a resynthesis engine that finds new synthesiser parameters to create an exact sound-alike and then deliberately recreate a piece "In the near style of artist X". With a little musical processing I can change the key, inversions, re-template the rhythm to a new swing... always pushing the derived piece into new territory until eventually it's barely recognisable.

Nonetheless, in the second case I have clearly intended to steal someone's idea and "make it my own" by automatable transformation.

In the former case it seems to be a "genuine labour". (whatever that means in 2023)

The genuine artist intends to make something through intellectual labour.

Maybe that's the real question. I mean, about "labour". If the cost of the labour tends to zero, does it really matter?

> There’s no technical reason you can’t back-track the weights and make a list of which tokens from which training data were sampled.

The weights map inputs to outputs, not training samples to outputs. They are adjusted during the training phase and then kept fixed (until there's a new iteration of the model; and let's ignore online learning for simplicity's sake now, the major models we're talking about here don't use it afaik). From the perspective of a user who inputs prompts and receives answers, the weights are constant. We could create some score indicating how much each training sample contributed to the weights overall, but then that score would be the same for all prompts.

It's really not clear to me what you mean by 'sampling from training data' for a specific prompt. The best I could imagine would be creating another metric measuring the distance between the prompt and all training samples, and then somehow combining that with the first score to get an overall contribution metric for this prompt. But that would be a) quite a bit of guesswork with a lot of modelling freedom and b) quite different from what you described.

The model isn’t sampling from training data during inference. It’s sampling from a latent space created during training, which used all of the training data to create it.

Fundamentally, the presence of a next token in some of the samples in the corpus is just as informative as the absence of that token from other samples. You can’t cite one without the other.

If you wanted to give attribution you’d need to list the entire training corpus.

> It’s not emitting “code”, it’s emitting individual tokens ... The “code” is the arrangement of those tokens, but that is determined by the weighting of the whole network

This theory of operation is not borne out in reality. It's been clearly displayed that these tools are emitting verbatim copies of existing code (and its comments) in their input.

It's even being seen in image generation, where NatGeo cover images are reproduced in their entirety or where stock photo watermarks are emitted on finished images.

And so, what can be traced back to individual sources? Quite a bit it would seem.

The output may be recognizable by humans as being “the same”, but an ML classifier can look at what is “obviously” a cat and classify it as a turtle. So while your statement is accurate, it doesn’t explain how “traceability” could really work—what is the actual mechanism of “tracing”?

There's a whole field of work around AI Observability - tracing how data moves through a model. It's intended to help identify biases in the input data, and to help answer the question of "why did it make this decision".

And if you can trace how data moves through the model, you can identify "that output came from these inputs" and add the associated metadata.

On your image generation point, something potentially interesting I saw on twitter/Reddit was about artists uploading “no to AI” images to artstation and then that logo almost appearing to vandalise AI generated art. I’m not sure if it’s a real thing or just people on twitter and stuff who don’t like AI art trolling though. Mostly because I’m not sure how up to date some of these models are being kept.

That's a joke/fake

> It's even being seen in image generation, where NatGeo cover images are reproduced in their entirety or where stock photo watermarks are emitted on finished images.

Can you cite sources? I’ve heard this claim repeatedly but have yet to see a good example.

https://youtu.be/kqPKNksl9hk?t=80 <- NatGeo cover

https://news.ycombinator.com/item?id=33061707 <- Watermarks (there's been a few linked here on HN)

The AI learns to generate watermark-like things, because those exist on a large fraction of its inputs.

It doesn't mean the rest of the output existed in the training set. It's perfectly capable of generating a completely novel picture, then slapping a watermark on it.

My point is that the watermark is deterministic. It is taken whole-cloth from an input, and reproduced as-is on an output. Thus, it is able to be attributed.

Stable diffusion is more like a paintshop artist who grabs bits and pieces of other art and melds them together, and less like a painter who creates from their imagination.

> It is taken whole-cloth from an input, and reproduced as-is on an output.

I respectfully disagree with this claim. It’s not as-is. It’s remarkably similar. That’s a big difference.

> Stable diffusion is more like a paintshop artist who grabs bits and pieces of other art and melds them together, and less like a painter who creates from their imagination.

I also disagree with this. The uncomfortable truth, imho, is that what Stable Diffusion does is FAR closer to what human artists do than we’d like to admit.

Human artists are perfectly capable of reproducing copyright infringing images. They just generally choose not to for legal/moral reasons.

> It is taken whole-cloth from an input, and reproduced as-is on an output

This is only possible for elements that are repeated hundreds or thousands of times. It's generally considered a failure of training data curation, if it's anything more than watermarks. The AI can learn how to reproduce the watermark, while being literally unable to reproduce the pictures it was on.

Neither of those are verbatim copies, do you have an example of a verbatim (i.e. exact, non-transformative) replica?

Well yes, if you’re asked to memorize someone’s code, you’ll get surprisingly far too. The fact that models can do this isn’t evidence of anything. It’s a capability (or “tendency” if you overfit too much).

I think it’s pretty obvious that if you train a model to reproduce existing work in its entirety, it fails the “sufficiently transformative” test and thus loses legal protection.

But there’s nothing stopping you from re-coding existing implementations of GPL’ed code. I used to do it. And your new code has your own chosen license, even if your ideas came from someone else. Are you sure the same logic shouldn’t apply to models?

Are you sure you can do that? Because it is trivial to paraphrase code and text.

Sure. I’ve done it for Bel.

  (def no (x)
    (id x nil))

  (def atom (x)
    (no (id (type x) ‘pair)))

  (def some (x f)
    (if (no x) nil
        (f (car x)) x
        (some (cdr x))))

  (def all (x)
    (if (no x) t
        (f (car x)) (all (cdr x))
        nil)))

I don’t even have to pull up bel.bel to know that those are almost perfect replicas. I typed it on my iPad.

EDIT: as far as I can tell, the only diff is that all comes before some. https://sep.yimg.com/ty/cdn/paulgraham/bel.bel?t=1595850613&

I could have kept going through most of the implementation.

I'd wager that most folks would consider the code for some basic list operations to be a little different from outputting a complete specific implementation of a fast square root function with its exact constants, including the comments.

Why? They’re both basic functions. I’d argue that if pg had left comments and constants, I’d have replicated those too.

Perhaps my example isn’t as impressive as the model’s capability, but operationally it’s the same.

> But there’s nothing stopping you from re-coding existing implementations of GPL’ed code.

There is if you've read the code. You can still violate copyright if you're hand-copying an implementation of a feature.

Lawyers recommend "clean room" re-implementations when reproducing copyrighted functionality (when folks bother to ask) precisely to avoid the risk of code being copied verbatim.

Functionality can be copied. Code can not.

> It's been clearly displayed that these tools are emitting verbatim copies of existing code (and its comments) in their input.

Which makes sense when you consider that the sort of code that is getting reproduced verbatim is usually library functions which developers may copy and paste verbatim comments and all into their project, especially when you prompt the AI with the header of a function that has been copied and pasted often, so the weightings will in that instance be heavily skewed towards reproducing that function

So that should make it easy to attribute, yes?

Probably why, like the article says, they're planning to add that

> In an attempt to address the issues with open-source licensing, GitHub plans to introduce a new Copilot feature that will “provide a reference for suggestions that resemble public code on GitHub so that you can make a more informed decision about whether and how to use that code,” including “providing attribution where appropriate.” GitHub also has a configurable filter to block suggestions matching public code.

I think harder, as it is spammed around in all directions. It's easier to attribute a unique piece of code that appears in a single repo.

But boilerplate functions don't deserve copyright protection as they are not creative. Can I copyright print('hello world!') if I post it in my repo? Do I deserve a citation from now on?

For better or worse, AI is a combination of machine learning algorithms. And these algorithms are black boxes solely because we don't add observability to them - we aren't looking.

But there is a desire to understand why an AI provided the output it did (to increase trust in AI generated output), and so there's a lot of study and work going into adding that observability. Once that's in place, it becomes pretty straightforward to identify which inputs to a model provided what outputs.

I have never seen an ML researcher claim that understanding the effect of specific training inputs on outputs is straightforward given the size of these LLMs. Most view it as a very difficult if not impossible problem.

And yet it's a major part of the overall concept of being responsible with our use of AIs. Throwing our hands up in the air and prematurely declaring defeat is not an option long term.

It's a non-starter for no other reason than potential copyright infringement means the government becomes involved, and they will stomp on the AI mouse with the force of an elephant - the opinions of amateurs and the anti-copyright movement notwithstanding.

As such, AI Observability is a problem that's both under active research, and the basis for B2B companies.

https://censius.ai/wiki/ai-observability

https://towardsdatascience.com/what-is-ml-observability-29e8...

https://whylabs.ai/observability

https://arize.com/

Those companies are not solving the problem you are describing

"No they're not" and "no it's not" (simplified from the actual response) are conversation enders, so I'll follow the lead and let this conversation end.

Have a great new years!

Well it helps to be accurate in your statements

Observability is great but it doesn’t give granular enough insights into what is actually happening.

Given a black box you can do two things: watch the black box for a while to see what it does, or take it apart to see how it works.

Observability is the former. Useful in many cases, just not here.

If you want to know what LLMs are actually doing, you’ll need the latter. Looking at weight activations for example, although with billions of parameters that’s infeasible.

Even if that theory of operation were true, it seems like an implementation detail that would be largely irrelevant in a court of law. Infringement is infringement regardless of how complicated your method for committing it was. If your AI reproduces a copyrighted work verbatim without permission, surely you are committing infringement? If you would be infringing by doing it via copy and paste, I would think a court would be likely to find that you infringed when you used an AI to do it. It does not matter that the algorithm on the backend is fancy...

Personally, I agree with you. Others do not.

But irrespective of legality, my point is that attribution is not such a hard problem as people believe, because their mental model of what's going on is much more complicated than reality. I feel they are seeing it as a true AI (mostly non-deterministic and thus hard to observe) and not ML which is deterministic.

Do you have a source for this claim? This isn't really how generative models work

https://news.ycombinator.com/item?id=34187038 points to the two specific claims WRT stable diffusion. The co-pilot ones are easily found. Quake 3's "fast inverse square root" is the best known instance.

The quote does not make that assertion…

This quote seems to fundamentally misunderstand what transformers are doing at all. Technically I suppose you could save all gradient updates from every input token, and do some weighted averaging to show what inputs affected the particular output the most, but saving all those gradient updates would be unimaginably space consuming. "Feasible" is doing a lot of work there.

It's very hard for people to get away from the idea that GPT is "copying" something, but that's not what it's doing. The reality is, to get the exact artifact which produced the code in question, you need "Call me Ishmael" from Moby Dick just as much as the Linux kernel source.

> The reality is, to get the exact artifact which produced the code in question, you need "Call me Ishmael" from Moby Dick just as much as the Linux kernel source.

Not always. Sometimes it just copies code without modification.

It never tells you when it does that, though. So to be on the safe side, better assume that it always does.

Even then, the presence of a next token is just as informative as the absence of another during training. That information also gets backpropagated. And during inference, good luck identifying which weights were responsible for a given next token (assuming we’re using greedy decoding, don’t even get started on beam search) let alone which samples contributed to that weight (hint: they all did).

It sounds like nonsense. The most plausible solution would be to provide credit to every single author whose code was used in the original training set; of course, that would run into gigabytes just for the credits.

And…? I think my last Xcode update was 20+ Gb.

Personally, I don't want minimalistic programs to start taking 20+GB! Xcode's bloat isn't a goal to strive for.

If such a giant list of acknowledgements were to be included, would it even help anyone, in any way? It sounds like pure inefficiency.

But GPL requires the resulting derivative code to be open-sourced under GPL, not just attributed. So if you did that then copilot could only emit code under a GPL license. Maybe that is okay, but it is not what copilot is trying to do at the moment. (And not everyone wants to open-source all their code under GPL.)

In the ideal case the next token is determined by the local context (the prefix string) and the entire corpus of trained code. In this case the prefix string has not been seen before and so the generator must do some interpretation/extrapolation to determine the likely continuation. But in some cases, perhaps many cases, the prefix string has been seen before, or is similar enough to what has been seen before, that the best continuation is just to spit out the similar string in the training corpus. Presumably such cases can be detected due to specific patterns of activation in the network and attributions can be captured/applied.

One dumb way to do this would be to include self-attributions directly in the stream of training data. So in the cases where the best continuation is to just transcribe the training data, the attributions is included in the data itself.

That “presumably” is exactly what I’m trying to get past. How would it actually work? Once the input corpus has been chewed up and reduced to tokens, is there actually a representation of “the similar string in the corpus” any more?

I don’t think people are even clear on the problem statement here. If I fed in three very similar functions from different sources, and I got a fourth, also very similar, output, what is the “traced attribution” supposed to be?

There's a clear difference between an output string that matches 10's of tokens from training data and one that is synthesized from the statistical regularities of the training corpus as a whole and so will not match any input data past a few highly informative tokens (ie ignoring structural tokens). The computational dynamic of doing synthesis should be noticeably different from doing verbatim copying. For one, the statistically relevant context for doing synthesis is much larger than doing verbatim copying. The prior 5 tokens may be sufficient to determine that the continuation should be the verbatim copy (eg quake's fast inverse square root). This difference must be represented in the network activation patterns in some form or another. I don't know what this difference is, but I know it's there.

The case of genuine synthesis is the best case for code generators and aren't the cases where people are expecting attribution.

I see, so the hypothesis is there's some way of determining "how much choice" it has for each output token. The less choice it has, the more we will call what it's doing "verbatim copying". Once the threshold is crossed and you want to "trace" it, though, I'm still not clear on how you can map back through the latent space and figure out where in the corpus it's copying from. Other than the technique suggested upthread, where you just literally search the corpus as a separate operation.

Honestly, I would be fine with verbatim chunks of code if it could gaurantee a compatible license and did the copyright notices properly.

That always seemed like a problem with open source, you should theoretically be able to copy bits of code from hundreds of projects to make a new one, but keeping track of the licenses makes it too much of a pain. So the closest we really see is people vendorizing libraries.

You can just find the most matching snippets in the training data using an embedding model.

They probably mean using a code search engine to check all snippets. The simplest thing would be am n-gram filter. A more advanced approach would use a code similarity neural net. It's not principled attribution, just locating the most similar example in the training set.

> It's not principled attribution, just locating the most similar example in the training set.

This is a profoundly important distinction.

Back-tracing data to contributory training examples is a genuine "influenced by" relation. Picking the nearest neighbour to a given result (even if its an exact copy!) cannot say anything useful with respect to origins. And given that there will always be some proximate neighbour, it's really a "misattribution machine".

This is bit like how our broken patent system grants or denies ownership of a design based on similarity to extant art but regardless of actual originality.

So there a few ways this can actually be done with different levels of accuracy/precision. But it is going to be complicated no matter what.

The easy thing to do is to compare outputs to inputs. This isn't technically hard (i.e. cosine similarity) but it is computationally difficult (i.e. cosine similarity of output to entire input). This would give us some weightings that show similarity. But this doesn't really tell us attribution, rather more a correlation. These are statistical models so there's reason to believe that this is okay.

Then there's inversion. This is processing data backwards through the model. This has different complexities compared to what type of model you're working with. GANs aren't great, diffusions are okay, normalizing flows are trivial (but good luck generating good images from NFs). If we can invert the model it is much easier to investigate and probe for contribution by looking at the distance of the latent generative variable to the location of the latent trained data. Basically you're looking at how different information is contributing to the overall output. This can also be done at every level in the network. Obviously this gets both technically challenging as well as computationally.

Another method would be using dataset reconstruction (this is outside my wheelhouse fwiw). This is where you try to recover the dataset from the final trained network. This too is complicated but there's plenty of papers showing progress in this space (lots of interest from privacy groups).

(TLDR-ish) There's other methods too. But basically what is being said is that there are ways to denote what and how much the training data contributed to the output of the model wherein we can then measure how similar the output is to the inputs (i.e. copying).

The answer is of course it's possible, so long as you have a couple of gigabytes spare for the acknowledgements page. Any construct worth attributing will have roots in billions of parameters.

It's far worse than that: to attribute a particular output to an input you don't just need the input data, you need the gradient updates that data caused in the training run. A couple hundred gigabytes input tokens times 175 billion parameters equals... impossibility.

Theoretically possibly, technically challenging. The training data would need to be annotated with traces, so a person would have to figure out where all of the answers came from

Maybe like a science paper... 10 lines of code, 50 citations :)

Reply on news.ycombinator.com