That’s not how the median works. If we had as many in poverty as billionaires, we’d have a pretty good situation.
Median and mean both compress n data points into one, which is only useful given some assumptions about the distribution. Quintiles (or similar) would work for that scenario, but I'm sure there are more complicated situations where they fall short.