Training open-source LLMs on ChatGPT output is a really bad idea.
gist.github.com
gist.github.com
>For this type of interaction, we must use RL training, as supervised training teaches the model to lie. The core issue is that we want to encourage the model to answer based on its internal knowledge, but we don't know what this internal knowledge contains. In supervised training, we present the model with a question and its correct answer, and train the model to replicate the provided answer.
The author says he’s summarizing a talk by John Schulman of OpenAI [2] but I haven’t personally watched the video. In any case, this is an interesting insight.
Say we set up a supervised learning scenario where we ask the model to use its internal knowledge to answer a question and compare its answer to one written by a human. If the two answers essentially say the same thing, but in different words, in the supervised learning case the model is penalized. In the RL case, it’s rewarded. That’s the difference.
1. https://gist.github.com/yoavg/6bff0fecd65950898eba1bb321cfbd...
However, my point goes beyond this technical argument. I would argue that even if, by some magical process, we could perfectly replicate GPT-4 behaviour, I still don't think it's a good idea or at least it's not enough. Don't get me wrong, it would be really handy to have a free version running on our own cluster, but it wouldn't fix the other issues I mentioned.
Though, as Go speculates, it's likely possible to reduce even further the HF ("human feedback") part, while still reshaping the model to have the helpful qualities.
My guess is there's a rich set of potential ways to this – automate that extra level of distinction between mere "exact token prediction" & "sufficiently valuable responses" – & OpenAI probably has a few undisclosed advances here as part of their GPT4 training/tuning.
In particular, Go's suggestion that a separately tuned LLM can do a fuzzier scoring of whether an answer is "close enough" to an idealized answer seems like the sort of promising ensemble approach that will have been an obvious next step for most LLM teams, probably being tried by many independent teams right now.
I see this pretty directly in stablelm releasing both a base model, and a tuned model… which is not based on the base ;)
There is a goldrush to get training sets worth using, and if something 90% quality gets your models on the map quickly, it’s an attractive option. As they say, attention is all you need.
Training in the 7B range is a lot cheaper than I expected. Fine-tuning almost negligible—if you have clean data, which has always been the expensive part.
Most humans expect more than $0.000003 per token as compensation for _your_ dataset collection.
I hate gpt style of "as an ai model I can/can't" answers, any model distilled from that corpus becomes very hard to use for tasking.
Like you may just want the category of a text, but all your equals now become contains. It eats up a lot of token space. It begins as a sentence so categories now are strongly biased toward sentence case and not your original input.
I know at least one model purposefully removing these utterances, but still. Everyone else is chasing the agent feel, and I'm here pulling my hair out because prosumer were this close to be able to access a proper AI for tasking and now it's slipping away.
Google doesn't have their own LLMs? Microsoft isn't running and training their own copies?
Hell, if you have enough money you could run your own. This doesn't sound like a monopoly in any definition that I use (and I cast a very wide net with my use of monopoly).
One thing he didn't mention though is that there's potentially a bit of a trick to get the fine-tuning datasets to transfer across models anyway. (I haven't tested it.)
The key idea is to eliminate the pronouns. Imagine asking GPT-4, not whether it knows a fact, but whether "gpt-3.5-turbo" knows, or "text-davinci-003" knows, etc. Then, when you want the model to reply using pronouns (e.g. "I don't know"), use the system message to tell it which model it is.
This doesn't benefit from introspection, so quite possibly it doesn't work. The reason it might work anyway, though, is that estimating the difficulty of a question might be possible even without introspection.
You can do the same with pure supervised learning and no RL. HF is the key, not RL.
Yoav misses the nuance John had. RL is not bringing something fundamental to the table. It is just a better way to do things at the moment
To be fair, we're already there, and we've been there for at least 10 years now. I'd wager >75% of the internet is garbage: auto-generated blog posts, programmatically-permuted ads, YouTube videos that mainly regurgitate other sources. Email is mostly garbage and the only reason it's usable is because spam filters have gotten pretty good. Even non-trivial amounts of heavily-curated social media (Twitter/FB/IG) is purely spam.
Some product comparison websites also seem to be built based on automatic sentence generation from tables with specs.
It's not the same as buzzfeed style content farming but it was a sign of the things to come.
For some reasons I never bookmarked those sites, when I left them in a rage and disgusted about so much information garbage. So it definitely has become way worse, but also 10 years ago I remember that pattern. Most often when looking for alternatives of software, then you were always a click away of being on a nonsense site, automatically filled with all the relevant keywords and lots of things to accidently click on, but nothing useful. Some of it might have been manual edited, but for the most part, I could allmost see the algorithms that filled those sites up with "content". There were just really primitive - so things will get interesting when this will gets combined with LLMs big scale.
Misinformation and manipulation is based on a small number of posts being shared and upvoted en masse, so AI won't help there.
However, social hacking and fraud involving actual dialogues with people is currently labour intensive and low yield. AI will definitely enable more of those attacks to happen automatically; and conversely, also help anti-fraud companies create puppet accounts to waste the fraudster's time, and thus the game of cat and mouse continues.
It's relatively easy to detect when someone is trying to sell you something. Does not matter if it's a custom written text or not. We had the true spam filters going 2 nines accurate already. (Google's is somehow misgauging spam or not learning my particular mix of spam well.)
Open question: is there any case where LLMs can be used for malicious purposes, but LLMs can't be used to defend against it?
It will be fun to watch the arms race where the spam generator need to conceal prompt injection attacks meant to circumvent such filters while at the same time be too subtle for a humans to pick up
There are approaches such as Dolly to develop a non-openAI RHLF feedback set but it's hard to compete against ShareGPT and co.
There are now uncensored open source models. Vicuna like models are great, and even work for translation. It's eerie what a 10GB file can do
The rate at which AI can generate text will be so much greater than what humans can generate.
You're already assuming pagerank and upvote systems won't break down in the future.
Presumably at some point computers will become (already are for all I know?) the largest consumers of content on the internet as well as its producers.
Now go ahead and spend $50 dollars on AI generated text nobody is ever going to read, just like almost nobody is going to read this comment.
Note that GPT-3.5 and above are already intentionally polluted with their own output by the RLHF process.
i'd say llm's represent a institutionalized reinforcement of bias (much like journalism) combined with some in-human autonomy.
So, not like a watermark, which would be impossible.
But: very fragile, especially if people are specifically trying to hide their GPT use, or have access to the watermarking algorithm or online oracle.
And: other methods – like remembering all output ever, or fuzzy summary representations of all output ever – seem to me similarly fragile, & introduce other problems & impracticalities.
A guess: OpenAI internally initially shared the common concern that "consuming its own junk outputs" could be a problem. But their own experiments so far, private & public, may have convinced them it's not as much of a problem in practice as it seems in theory. The model outputs have a mix of good and bad text – just like the pre-LLM internet. And, the same filterings/weightings that have worked on pre-LLM content keep working. And, counter to some early intuitions, often one LLM's quality output is in fact very-useful input for other later LLMs.
(You can skip to the section “Verifying“)
But before its concern about a monoculture, the article 1st points out that mere prediction-training (on another model's outputs or fresh data) can't truly match RLHF in instilling some much-desired behaviors.
And that presents a bit of a tension with the article's 2nd concern: if mere output-mimicking *can't* match more-sophisticated training, then it can't really create the concerning uniformity, either.
And maybe: the dose makes the poison. A little cribbing might be a beneficial partial accelerator for smaller teams & newer projects, even if a lot is ineffective (at ever fully replicating OpenAI model behaviors), or deleterious (if effective and also overdone).
So the article isn't really a strong case for not trying this at all – just for keeping the potential limits & downsides in mind, in any experiments with this technique.
The reasoning that the results won't be quite as good as RLHF, or result in a perfect 'clone' of ChatGPT's capabilities, seems pretty good to me.
But the idea it won't be helpful at all, especially to projects that are just seeking some incremental advantage? Seems speculative.
In particular, when you read the linked comments from Yoav Go, he outlines a potential RL process that uses automated scoring for non-exact similarity to preferred answers. Using known (or even 'probably') good answers from ChatGPT output, as the inputs to that process, seems like it could often offer some of the same sort of improvement to other models as ChatGPT obtained via its RLHF.
A subset will give you a subset of the knowledge, it’s no free lunch
Some recent research hash showed reasonable success transferring fine tuning between models using outputs.
When I finetuned successive generations of ESRGAN on its own output (as I essentially wanted to use it for img2img), it would amplify tiny oddities and artifacts that, I would later find out, were in the training data. Tiny noise splotches, "swirls" and distorted line edges blew up. And I was careful... I pixel peeped the dataset as best I could before starting training.
Human language is obviously different, but I still fear oddities or biases will start popping up when the base models train on large fractions of their own data. And by the time we find out, it will be near impossible to filter out.
But continuing the analogy, maybe a diverse base model population is a good way to avoid that issue?
The gulf between otherwise smart people on this very important issue should depress us all. Personally I feel as if a mutant species has been released into the wild, yet as in Rick and Morty, some people think the wisest course is to release a lot more mutants.
People are fools. Hackers more than most— though we are productive and useful fools much of the time— but it hasn’t been a threat to humanity until recently.
is there a how to anywhere - not even sure which opensource model to use, etc