Sure but are you standing on the shoulders of Stable Diffusion or not?
Fine tuning Stable Diffusion with your own images is way easier than creating Stable Diffusion in the first place.
If you're creating your own I stand corrected and that's some serious investment.