There are two problems with this simplistic (but yes, mostly good) view.
One is that those extra zeroes may have come from exploiting you, and reducing your potential wealth. This is way too common, so that many people assume it by default. It is getting less common, in a slow process lasting for centuries, but is still way too common to ignore. Those extra zeroes also lead to more opportunities to exploit you, in a vicious cycle.
The other one is that some goods are limited, and you will be in direct competition with those people on many markets. Ever wondered why Silicon Valley is trying to solve entertainment for rich people instead of basic survival for the homeless living there?
That said, yes, I would agree that people being rich is mostly irrelevant for not-rich people. But there is some dependency on the details, and at some levels it completely stops being irrelevant.