As an entrepreneur, growing a "runt" company is basically wasting your talent, and that of anyone else you have working for you. Your own skillset, and all the company's other assets, are locked away doing something that's not making anyone--especially you!--nearly as much money as you could if you tossed the current company out and rolled the dice again. If a larger company was responsible for the project you were managing, they'd shut it down and move you to something else where you could make them more money (see: Google Reader.)
In economic theory, this is why layoffs and recessions are good: they kill companies that were just getting by, and release the human capital "stored up" there to move to newer, brighter ventures.
I mean, it's fine if all you really wanted was a lifestyle business--but lifestyle businesses don't take investment.