The weights need to be connected to something interesting. Pre-training is how they get them all connected up, and fine-tuning is how they find which weights it would be useful to change.
Pretraining determines the weights (e.g. the connections), fine-tuning let's you change some subset of the weights (e.g. the final layers) with a smaller chunk of task-specific data.
It would be cool if there was like a threshold in training from scratch, where after this actually works for adding higher level knowledge. So you would start training it like usual to get it to absorb generic langauge skills and reasoning but save all the domain knowledge absorption for merging later
Isn’t x + 0 = x?
So what’s the benefit of starting with 0 when you can just start with x
I know it’s over simplistic above, but serious question, please ELI5..
And then you apply a different models weights using the papers methods and then another