And I knew the math beforehand. I was a Physics major in college with a CS minor.
As with most topics in software engineering I'd say you will be have to keep learning as you go. They keep coming out with larger models that require fancier parallelism and faster data pipelines. Nvidia comes out with a new thing to accelerate inference every year. Want to use something else than Nvidia? Now you need to learn TPU, Trainium, Meta Accelerator (whatever its name is).
IMO you can only learn al that by doing a few successful ML projects end-to-end. So, a few years?
Not that long then, especially if someone was motivated enough to complete the projects as quickly as possible.
Is this a catch-22 then or is there a rational course to self-study into the field for those that are competent?
Well, these were senior level skills, a person who can drive and complete a project. I don't know how you could become senior via self-study and without practical hands-on experience on a project (working with and learning from somebody with experience).