How big of a perf improvement would result from using the architectural tweaks that Llama3 and others have put in place since GPT-2?
1. delete the absolute positional encoding and replace with RoPE
2. delete all biases in all layers (in LayerNorms, they turn into RMSNorm)
3. GeLU -> SwiGLU non-linearity in the MLP
4. longer context length
5. architecture hyperparameter changes, e.g. slightly different aspect ratios
And there was a paper that I can't find the reference to anymore that claimed that if you train long enough, the gap becomes even lower. Possibly because the absolutely positional encoding has enough time to train more fully, where as the RoPE layer benefits from the "inductive bias" it adds in the earlier stages of training.
But I don't have full confidence on the above claim, maybe someone has tried or has better/concrete reference.
self.w2(F.silu(self.w1(x)) * self.w3(x))
https://github.com/meta-llama/llama3/blob/14aab0428d3ec3a959...