When used it's a description that the business owners provide based on how they run their business, then it's OK. When you use it to describe someone else's business it's often considered a slur (unless they've previously shared this with you and you know that fact.)
Many smaller companies are not actually lifestyle businesses although that term is thrown around here. The owners may be part-time because they actually operate a group of companies, grow niche companies to sell outside of the VC world, etc. They are indistinguishable from lifestyle businesses, but the owners are looking to do a lot more than sustain a level of income.