You've heard it from sensible people because there's a significant social-sciences literature studying the idea, and largely confirming it – no "dog-whistles" required. I want more ethnic diversity in the US, but my eyes are open that in practice, such diversity (for at least a generation or two, maybe longer) tends to weaken political support for public services, and strain such services' competent administration. The right cultural & policy innovations
might be able to overcome this effect! But probably not, if merely acknowledging it becomes a taboo, and we pretend it's not an effect at all.
And it's not just ethnicity – it's many dimensions of "largeness" and "diversity", including geographic dispersion, variety of economic industries and incomes, religions, density, etc.
Only 2 countries in the world have a larger population than the USA: China and India. They're not known for top-notch free public services, either – though China's anti-union, anti-local-control authoritarianism allows public infrastructure development at a crazy pace, and (forced relocation into) public housing. The other 7 most-populous countries (Indonesia, Pakistan, Brazil, Nigeria, Bangladesh, Russia, Mexico) aren't stellar examples of public services, either.
The "high services" models most often highlighted tend to be less than 1/30th the population of the USA and far more culturally/economically homogenous (the Nordics), or at most 1/6th to 1/5th the population of the USA (UK, France, Germany), and still less varied geographically, economically, and culturally.