You're assuming your family is entitled to keep forever everything that it happens to have received in its economic activities.
Elizabeth Warren famously said in her first Senate campaign, "There is nobody in this country who got rich on his own. Nobody. You built a factory out there—good for you! But I want to be clear. You moved your goods to market on the roads the rest of us paid for. You hired workers the rest of us paid to educate. You were safe in your factory because of police forces and fire forces that the rest of us paid for. You didn’t have to worry that marauding bands would come and seize everything at your factory, and hire someone to protect against this, because of the work the rest of us did." [0]
In his book The Economists' Hour, NY Times economics columnist Binyamin Appelbaum argues that free-market icon Milton Friedman "celebrated drivers and took roads [and cars and gasoline and ....] for granted." [1]
Some would doubtless argue that the "free market" takes care of all this via pricing. But the "free market" really isn't; companies strive mightily, and often successfully, to seize pricing power and use it to extract rents.
[0] https://newrepublic.com/article/95247/elizabeth-warren-class...
[1] https://www.amazon.com/Economists-Hour-Prophets-Markets-Frac...