If by diverse you mean characteristics such as ethnicity or gender/sex for example, then you're right. The value it (may) add doesn't justify optimizing my hiring pipeline for to increase their representation in my organization. After all, I'm not interested in building a mosaic of all sorts of people. I'm interested in meeting goals of my business. Everyone who can contribute to that effectively is welcome.
When we have a candidate who is from a "minority" group and we treat them differently because we assume they suffered more to get where they are so they paid their due and now are entitled to the job position, isn't that an unconscious bias too?
When we look at white men and assume they had an easy ride in life, isn't that an unconscious bias too?
Look, everyone belongs to different groups at the same time so they have more privileges and advantages in some areas, while in a minority group in another. How far should we go? should we also check who has the least privilege and representation within a minority group before we hire them? Do people in minority groups ever run out of privilege once they acquire it?
This whole idea of treating people differently based on their race or other irrelevant attributes is discriminatory and racist. How the hell did we get here? I don't know.