In both cases- ageism and racism, the employer culture is hiring less qualified workers based on criteria other than skill and competence. That necessarily means they have a lower caliber of expertise on average, as they are artificially decreasing the pool size of potential candidates. Moreover, if they have a discriminatory hiring practice in the first place, it is very unlikely that they are really selecting for competence even within the preferred demographic. For an extreme example, look at the current US administration, hiring only people with "Mar-a-lago face" combined with them needing to have a particular ideology and extreme lack of ethics, necessarily makes virtually all of the hires incompetent. They aren't, presumably specifically seeking incompetent hires, it's just the only people left given the fact that they select for things other than competence first.

I hire software engineers, and given all of the discrimination in the field, my willingness to hire women, minorities, and older people gives me a substantial advantage over those I am competing against that have discriminatory hiring practices. Although I'm not selecting for discriminated against groups, in practice I hire almost exclusively from them, because there are a surplus of extremely qualified people in those demographics that are getting rejected from everywhere else. I'm sure there are a ton of competent young white men in software engineering, but they're never among my top applicants, presumably because they've already been hired elsewhere.

This widespread discrimination pre-loads the applicant pool, such that the effect is enormous: you are getting substantially inferior employees if you have discriminatory hiring practices, because so many others do to, so your "preferred demographic" is already depleted of competent applicants.