Do you want to make a job out of your site, or building a framework for your site? These are two different things, and spending time one one tends to prevent you from spending time on the other.
That is true, probably build out the site and extend/fix the framework as necessary.
Isn't it the same thing though?
And if that framework is open source, and somebody else isn't extending or fixing it for you, you can do it yourself. You can still hack on it, but you don't have to build it from the ground up.
I respect building something from scratch so you can learn from it (I once wrote my own OS), but that tends to be helpful only once. After you've learned from the experience, you tend to be better at judging what to tackle yourself, and what you can leave to other people.