Wealthy people can purpose far more of their time.
Accumulating capital works like you say, bit we have work to do otherwise.
Improving basic wealth is possible, and does not impact capital much. Some, but not to a degree many, if not most will be concerned about.
It should be OK for ordinary people to be a bit wealthy. Arts, research, even just play are all rewarding, important.
For a reasonable amount of our time, yes. People can and should do what they want. It is not necessary we tighten this all up to some totalitarian optimum.