Inequality means stagnation of outcomes. You don't just want diversity for ethical reasons - not everyone cares about that anyway - but because by denying talented individuals the opportunity to develop, you're sabotaging future prosperity.

Of course you can argue that as long as you have enough talent this doesn't matter.

But invention and innovation are wildcards, and you can't always create gamechanging developments to order. If access to environments and resources that develop talent is heavily rationed, you're more likely to get a non-linear collapse in output than a linear decrease.