What is the main difference between training and fine tuning?
Can you start with a model trained only in producing the letter a, and then fine tune it to learn b, then c, then words, sentences, etc?
Can you start with a model trained only in producing the letter a, and then fine tune it to learn b, then c, then words, sentences, etc?
However, sometimes you can't do that. For example, perhaps you want your model to always talk like a pirate, but you don't have billions of words spoken like a pirate to train on.
So the next best thing is to train a model on all english text (which you have lots of), and then finetune on your smaller dataset of pirate speech.
Finetuning is simply more training, but with a different dataset and often a different learning rate.
Typically, finetuning uses far far far less data and compute, and can be done by individuals with a home PC, whereas training a large language model from scratch is in the $1M - $1B range.
What you are suggesting is called "curriculum learning", and though it hasn't been applied to LLMs yet to the best of my knowledge, it has proven to improve learning and decrease training times in other areas of ML.
I mention the "undoing RLHF" since it's not uncommon for fine-tuned models to increase in error in the original training objective after being fine-tuned with a different one. I think people saw this happen in BERT.
Also ChatGPT is almost certainly huge.
We have already documented evidence of the effect of this. In the GPT-4 technical report [1], they reported contamination of humaneval data in the training data.
They did measure against a "non-contaminated" training set but no idea if that can still be trusted.
Why would this matter? We can have seemingly strong benchmarks for containments but measures poorly against new and quarantined information. Classic over fitting.
Another argument is that data being put out there could very much be wrong and the amounts of it amplified by other models. Take a look at this sample of demonstration data for codealpaca [2]. Not only is its output wrong but bad practices like,making up a random computation without it having access to a place to run a calculation, teaches the model these type of responses are ok.
{ "instruction": "What would be the output of the following JavaScript snippet?", "input": "let area = 6 * 5;\nlet radius = area / 3.14;", "output": "The output of the JavaScript snippet is the radius, which is 1.91." }
1: https://cdn.openai.com/papers/gpt-4.pdf 2: https://github.com/sahil280114/codealpaca/commit/0d265112c70...
Useful for taking a generic model with a base level of knowledge, and tuning it so the output is more useful for an application specific use case.
- you could add new units throughout and train those while freezing existing units (adapter-based fine-tuning)
- you could train all units and use e.g. low-rank adaptation to limit how much they can change
- you could do prefix tuning and train an input to add at every layer
see e.g. - https://lightning.ai/pages/community/article/understanding-l...