That's why your typical middle-class earning $100,000 or more have difficulty building up the net-worth; that income is earned income, it is taxed much higher than passive income, and there's not enough time. I find it ridiculous to go for a 2 hour commute -- that is 4 hours each day, 5 days a week, or 20 hours. 20 hours of time spent in a car -- that's a part time job. A part time job you spend money, waste time and get nothing. (Even if you like driving, do you really enjoy driving during rush hour?) It gets worse when both couples in the family work, as now you're adding in greater expenses for working (a second car, clothes, lunches, and so forth). But the wage earners of families commute because they want their kids in good, suburban schools. Without time, there are very few investment vehicles available.
Contrast that with someone who have income-generating assets (high net worth -- primary residence is not an income-generating asset, and neither is the car, the SUV, or the boat) that meets/exceed expenses. If you don't want to stop working, you can keep working. Until your boss pisses you off, or you're asked to do something unethical. You don't have to commute unless you want to. If you want to go into real-estate as a landlord, you can focus on that without having to worry about meeting the next month's living expenses. If you want to do a software startup, you don't have to worry as much as burn rate -- just start coding. This appears to reinforce the cliche, "it takes money to make money", but it isn't. As mentioned in the previous paragraph, having money but little time means you're not going to be able to build up your net-worth.
And if you're pursuing a life-long passion, it doesn't matter if it is "active" leisure or "passive" leisure. It doesn't matter if it is stressful or not. The basic living expenses are covered. The lower parts of Maslow's heirarchy of needs are covered. The fact is, you're pursuing a dream. That's wealth to my eyes. That's liberty and pursuit of happiness, right there.