We were always in this regime. In general, it has been far more effective to take a proven model and fine-tune it. It was true five years ago taking a convnet out of a model zoo to fine tune and it’s true now.

If you can achieve the moonshot of gathering, generating, annotating enough data with the right distribution to train from scratch, and you use a SOTA bag of tricks to regularise, you might do better.

Bear in mind fine tuning is literally just more pre-training. Starting with a trained model is like starting with an incredibly well-initialised network.