I'm in a similar situation as the OP, and have heard that argument used for me. I don't like it. In order to live frugally and responsibly, I was lucky to have the upbringing that taught me to do it, and to not have anyone depend on me so that I could focus on getting to this point. I was lucky to live in a relatively wealthy country during a relatively stable time.
I made a couple of good decisions, and put in some effort, sure. But a lot of people work harder than me and make harder decisions every day, and they are doing a lot worse than me. What separates me from those people? Luck, mostly.