This is every nation state on earth. Of course they care about your background, skills and likelihood of naturalization. It's hardly unique to the US
How is all that relevant when you're on a week long business trip or visiting as a tourist? Vast majority of visitors in USA are not staying.
I am afraid they actually do. I know someone’s visitor visa was rejected because the consular officer thought he works on AI. I have no idea why working on AI is grounds for rejection.
It seems a bit non-straightforward that the right of a private club to choose members seems like an important freedom, but that that only applies when there are numerous competitive alternative clubs. For countries, that's usually the case but America is number one in many ways and sometimes by such a wide margin that applicants don't really have a choice that isn't much worse. So maybe it has more responsibility to be fair in order to avoid being tyrannical to the rest of the world.