Where does the "harm" you talk about come from? Usually it's because there are other responsibilities you're engaged with. If you literally give everything else up to focus on one thing, these barbs aren't in your life. I'm talking forgoing rent to crash with family/friends, eating ramen, selling/getting rid of your things, live on credit cards, etc, and have a 24/7 focus that keeps you going.
You'd be surprised how much that frees you from the immediate "harm" you describe, but you also have nothing to fall back on, because you pretty much have nothing else anymore than your pursuit, and can be amassing enormous debt (financial and otherwise). It's a huge amortized, pushed-forward harm in the hopes that the gamble pays off. For many, it doesn't.
All the things you described doing are the fallback options - you have to have a family and friends, things to sell, credit to burn. Even then, family or friends have less patience to carry you when you’re taking a moonshot unless they also have a lot they can fall back on.
That's not a judgement on what they choose, but many times the "lack of privilege" is really they because they made different choices.