The key issue is that money often translates to such things as power and leisure. Prosperity is not consumption - it is the command over power, resources and time.
The poor have to sell their time in order to afford the basic necessities of life; the rich don't have to. So the rich have a lot more free time than the poor and the resources to use it well. The rich simply are freer than the poor, who are not unlike prisoners with no claim over their time.
The rich also get to influence policies to a far greater extent than the poor. In a way, wealth is just stored influence. This in turn helps them perpetuate their privilege. For instance, they can fund narratives that normalize inequality and lobby for lower taxes.
The lives of the rich are also far more secure than the lives of the poor. Many poor people are one major life crisis away from penury. This significantly affects the quality of their lives. Access to more wealth would mitigate this.
One could also flip your argument as follows: wealth is a scarce resource. If the rich already have everything they need to live a happy life at low amounts of wealth, then letting them horde more wealth than necessary is unjustified. Instead, that should be distributed to those in need. This would make no difference to the well-being of the wealthy, but it would help others who need resources more.