I take issue with the use of "derogatory". I would be ecstatic if I could bootstrap a startup into providing myself with $100k+ per year, with no potential for substantial growth from there.
My impression was that the term also strongly implied that the business owner had the freedom of spending their time how they liked. I have a friend who lives relatively comfortably and does freelance development. If he wants to work 16 hours one day on a cool project and take a long weekend for some biking or hiking, he can. He isn't going to be a rich, rich man in terms of money anytime soon, but he is not wanting of much, either. He is happy, satisfied, locally respected, relatively stress free and content. That sounds like a lifestyle I would enjoy.