Almost by definition, a lifestyle business lacks the potential for massive growth. If it has it, and the owner tries to 'hold it back' someone else will come along and capture the rest of the market. The incentive to do so is large.
Occasionally, you will see privately held businesses that have the potential of startups, but they are not lifestyle businesses (maybe mailchimp). They grow into full fledged businesses that just happen to be privately held. They will often find ways of funding their growth (and have options for doing so), even if that isn't VC.
That said, lifestyle businesses are awesome for your lifestyle. I didn't think I wanted one until I ended up with one, and it turns out high-ish income, total control of your time, and direct positive relationships with customers are a great lifestyle for me.