Training these models from scratch on your domain specific data is not as expensive as one might think. We have provided some cost estimates in our blogs.
https://www.mosaicml.com/blog/mosaicbert
https://www.mosaicml.com/blog/training-stable-diffusion-from...
[1]: https://twitter.com/jefrankle/status/1649060478910357504?s=4...
As you note, with the plethora of open/open-ish LLMs today and LoRA + PEFT you can fine tune with low VRAM and pretty quickly so even a single A100 or whatever cloud GPUs are just fine. I've even seen people pull it off in reasonable time on super cheap T4s, A10s, etc.
I doubt anyone reading a blog post is attempting to train a "true" multi-billion param LLM from scratch.