It’s infuriatingly hard to understand how closed models train on their input

Cutting edge AI models and datasets have largely been "for testing" and "for research" ever since they existed. Usage rights were extremely low on the researchers' list of concerns.

But all of the sudden OpenAI and such are trying to commercialize models that (unlike a classifier or whatever) are very capable of stepping on rightholders’ toes...

Hence I suspect all the talk of trade secrets is just a front.

And then, years from now, test out new models by asking "What is Qwitzatteracht?" And if the AI respond with anything involving it being a golf game, then... I'll know. I'll know my prompts were used for training their model.

Because I could see someone taking a loophole and analyzing data collected and determining that 'well, it is OK to train on prompts that did not include identifying information or private user data' and see the above simple prompt and it be classified as 'not private data' and then get shoveled into a model during a round of training.

A non-answer for you, but for curious others, [State of GPT] 10 days ago provides a through introduction to the process used to train, Karpathy speaking at a Microsoft event providing a deep summary review of concepts, training phases, and techniques proving useful in the world of Generative Pretrained Transformers.

[State of GPT]: https://www.youtube.com/watch?v=bZQun8Y4L2A

This is such an interesting time - when the rules of the game haven't been settled. It was like when Uber was openly flouting established laws with their service. They knew that new laws would be required eventually and so they were willing to push boundaries on ambiguous interpretations until the matter was clearly settled.

What surprises me is the stance people take in the face of ambiguity. For example, when I see such vague terms I assume that the reason is so that github has the plausible deniability in using the data for things like model training. I don't mean to say it was written explicitly so that they would be able to use it for that purpose. I mean that the ambiguity, and lack of a desire to clarify it allows them to operate for some time with the plausible deniability. A product manager could make an internal case that a temporary competitive advantage can be realized by interpreting their terms of service in a particular way.

Eventually, law is going to catch up. And when it does Github, Microsoft, OpenAI, and all like them will follow the rules. But I, unlike the author, just assume that they are using this window of ambiguity to use the data they have available to gain an advantage. Maybe I am just cynical and the author is more trusting.

The truth is probably somewhere in between. They are probably using more than many would feel comfortable with, but probably less than my worst fears.

That paper (in January 2022) says:

> To make our models safer, more helpful, and more aligned, we use an existing technique called reinforcement learning from human feedback (RLHF). On prompts submitted by our customers to the API[A] our labelers provide demonstrations of the desired model behavior, and rank several outputs from our models. We then use this data to fine-tune GPT-3.

This was before they introduced their policy that data submitted to the API would not be used in any way to affect future training (that was in March this year).

Personally I think there's a significant difference between "your input to the model will be used as raw input for future pre-training" and "your prompts to the model will be used in an exercise where human labelers are shown multiple responses and asked to pick the best one".

What I'd really like here is for OpenAI to help people understand how that data is being used in as much detail as possible.

I guess the problem with that is it makes it harder for them to use that data in new ways they haven't yet anticipated needing in the future.

I've found it helpful to think of AI models as if they're people. What does that mean? It means when I'm talking to them I keep in mind that I don't know how they know what they claim to know, I don't know who else they're going to talk to or what they're going to repeat that I told them, that there's no guarantee they'll even try to treat me kindly. Trust AIs as much as you trust a random stranger on the bus.

Instead, I believe they use logs of user chats to detect when the user was dissatisfied with the AI's response (for example by detecting closing the window). They'll then pay a human to review these conversations, and if a human can write a better response, then that will go into the next round of training.

As others have pointed out, the secret for the NN may be to just have as much parameters as possible. So if I had a ton of money to train a LLM, I'd spend that money on curating a corpus of text. First priority would be books and papers out there. Then, maybe quality information on the internet like Wikipedia, Stackoverflow, source code, documentation.

After all, the low-level functions like grammar, sentence structure, word use can be learned from any good text. For higher-level stuff, I would want quality input in there. So it learns reasoning as scientists do in papers rather than some politically biased person on Twitter.

(We've seem some old emails of CEOs posted online. Something like this could be of value because they discuss strategy. But these are the exception. I guess even for interesting public figures, most of the conversations wouldn’t help much. It would also put OpenAI in danger of leaking sensitive information.)

For instance if I have some body of text that can't be found elsewhere on the internet, if the reply of the model references the information in that text in some way you may be fairly certain it was used in training.

The hard part is probably finding such a body of text.

Outputs often repeat the inputs (in fact, this is one of the behaviors that reinforcement appears to intentionally increase).

So: You provide a secret. Raters prefer outputs with repeat your secret. Model becomes more likely to say the secret thing, even when its not in the prompt.

> Submissions will be double blind: reviewers cannot see author names when conducting reviews, and authors cannot see reviewer names. This means that the submission must not contain acknowledgements or any link (e.g., github) that would reveal authors’ identity.

While Chinchila certainly is not a github link it is an acknowledgement that reveals the lab identity. Which leads us to one of two situations

1. The reviewers don't know the model/dataset is proprietary: In this case, we likely have a reviewer that is unqualified to even be reviewing.

2. The reviewers know: In this case, there's an ethics violation.

== Why this is a problem ==

There's several issues here: bias, evaluation, reproduction, ethics/scientific goals.

Bias: Most of these models and datasets are well known within the community and thus we know which labs they are from. We're biased to believe that works from big labs are higher quality and more accurate. This may be true to some extent, but your purpose as a reviewer is to judge the merit of the work. We try to do so in as much isolation as possible to prevent ourselves from being influenced and just focus on the merits[note0]. For example, any work with JFT is going to be known as a work from Google.

Evaluation: We'll use this work[2] as an example (it uses JFT and a lot of TPU compute: i.e. Google). Since the work only uses JFT we don't know if this scaling only works for JFT trained models, especially since it depends on the pretraining not overfitting (I'm unconvinced tbh)[note1]. At least their table compares mostly to other JFT models, but it is mostly meaningless because JFT changes size frequently and we can't assume consistent data. All we know is that this result is valid for JFT pre-trained models. Their claims are much stronger than this.

Reproduction: This is rather obvious. But we also have good reason to believe that these models change as well as the datasets. So even if there was a leak we have issues with works quickly losing validity as the next work is using different settings than the previous and intimate knowledge would be required to know if this affects results or not. NLP has this issue with OpenAI changes. It is very noisy research doing things like this.

Ethics and Scientific goals: Our job is to advance human knowledge. This can be done in a noisy process, but we do have a responsibility to make this as clear as possible. Our job is NOT to be ads for big companies, but closed work does so. In fact, this is a big reason big companies publish. They are great publicity. This is fine, as long as they provide utility to the community. We have other ethical issues[note2]

== Proposed Solution ==

Submissions must utilize open models and datasets. The work is evaluated based on that. After acceptance the works can add. This way there is no deanonymizing information, we ensure reproducibility, but we also are able to gleam the potential results from the proprietary efforts. This allows for the big labs to get their publicity while ensuring that anonymity is not broken and we're not biased to big labs. This doesn't solve the arxiv issue and the dumb social media policies[note4] but it is a a simple step we can implement right away. It is also a step that YOU the reviewer, a single person, can take action against. We need to take a serious look at ourselves and ask what is the point of our science. Is it to make money or to progress human knowledge? You can do both, but what is the main goal. I'd argue that our goal is knowledge. It is fine to make profits off this knowledge, but I'm afraid we've incentivized the profits too much and our system is not aligned with the original goals.

Goodhart's Law is a bitch.

== Notes/Bib ==

[note0] There are some biases we should concern ourselves with, such as compute power, but this requires more nuanced understanding. A work that can spend months tuning hyperparameters will always have better benchmarks than works that spend a month of compute for their entire work. These can't be evaluated on benchmarks alone and doing so will cause us to miss extremely important works (see Lottery Ticket Hypothesis[1]). Reviewing should be hard and require nuance, so there should be certain roadblocks to make it difficult to be lazy.

[note1] They do near de-dupe but we can't know how effective the techniques are so we can't know how valid these claims are.

[note2] Conferences are a bad way to do science. Conferences are filters, journals are QC. ML/CS uses conferences. This is a zero sum environment where your competitors are given the opportunity to filter your work. You don't need active collusion to form bad actors. It pits the people evaluating the work against the work. A journal system instead has reviewers focusing on adding value to the work. This is because journaling has rounds and discussions. Conferencing is nearly zero shot with a single chance for rebuttal, often limiting to a single page to respond to 4 reviewers with vastly different concerns. We have other issues like reviewer quality (remember ACs oversee reviewers[3]), qualifications, collusion[4,5], fraud, and more. But these are for a longer discussion.

[note3] Social media rules prevent authors from evangelizing their works in public, such as on Twitter. It does not prevent doing so in private or others from doing so for you. This not only leads to (actively encourages) collusion rings but worsens the problem. Big labs will always get extra publicity. People watch for their works, they have more followers, and so on. Small labs' only chance to get heard is to actively evangelize their work, and we just cut off their tools while not affecting big labs. A scratch to the giant is the loss of a limb to an ant. We shouldn't need evangelization but networks exist so it is necessary. I don't think we should stop pre-prints (they're too valuable), but we need to recognize the problems that they result in. We also shouldn't have people at the top of the ivory tower making decisions about how the handle the plights of those at the bottom without actively working with those at the bottom.

[0] (Notice Carmen's review) https://openreview.net/forum?id=OpzV3lp3IMC

[1] https://arxiv.org/abs/1803.03635

[2] https://arxiv.org/abs/2305.13035

[3] https://qipeng.me/blog/what-does-an-area-chair-do/

[4] https://www.reddit.com/r/MachineLearning/comments/nxbcl9/d_c...

[5] https://analyticsindiamag.com/whats-wrong-with-the-paper-rev...

FOSS, and openness broadly never made traditional business sense. But... success creates new types of business sense. By 2000, the early 1990s idea that AOL could have beaten the WWW was "OMG! Remember those naive boomers!"

The power of openness was proven by some point. Google, among others, created an empire in that world. Everything they did was WWW, lived on linux and they outpaced MSFT like a new breed. The Fact that Open was powerful was plain. The only questions were why and how.

Wintel had beaten Apple as a "platform" because it was more distributed. More Open = more distributed.... that's why Google and WWW were smashing MSFT. The Cathedral and the Bazaar. Information wants to be free. Billion dollar industries will give way to a million dollar side gig.

It's easier to weave these threads when external reality has already confirmed the conclusion. "Traditional Business Sense" about IP ownership, control, secrecy, competition and such was a dinosaur. It never went away though.

Times change. Culture changes. Business landscapes change.

OpenAi is, because of the name, almost comical. One apparent driving factor is that being closed helps shield OpenAI from criticism, liability and whatnot. simonwillison is more or less demonstrating this here.

Brainist?

Artificist?

Intelligencist?

Biologist?