As someone who did the same, most people do not have the privilege of having an income where they can just quit working and live off of savings before retirement. If they have a college education or home, they most likely have loans that need to be paid and that ties them to a job.
> Am I not supposed to do that at all?
No one is suggesting that. We're just particularly lucky to be in this position in the first place is all.
The privilege argument seems to come out when people like us lose that perspective and expect others to do the same or are shocked when others won't. I don't see this applying to you at all because
> I'm not going around telling everyone to quit their job and be as ostensibly great and care-free as I am.