That's true for some reasonable definitions of "privilege" but not others.
Originally-originally, "privilege" meant being subject to some special dispensation that e.g. exempted you from laws that applied to everyone else. (Latin: "private law".) The term then broadened to apply to other sorts of advantages held by individuals and groups, and indeed earlier uses of it are concerned only with unusual advantages enjoyed by (say) the upper classes. But these days it's commonly used to mean any substantial unearned advantage that people have as a result of belonging to one group rather than another.
And by that definition, a kind of privilege absolutely can be widely distributed.
For instance, it is widely and plausibly held that being male confers substantial advantages in life -- that people take men more seriously, are more inclined to promote men to senior positions, tend to give credit to men rather than women for projects in which both are involved, etc.
(In, let's say, Western Europe and North America.)
Some people disagree with that, saying that maybe it was so 30 years ago but that that sort of sexist discrimination has basically gone away now. Some of those people go further and say that actually now it's being female that confers unearned advantages.
I'm pretty sure that almost everyone would agree that at least one of the following two propositions is true: (1) 30 years ago, being male conferred substantial unearned advantages in life. (2) Now, being female confers substantial unearned advantages in life.
Either way, we've got an example of a widely-distributed characteristic that confers substantial advantages for reasons other than merit.