I think the claim of people who talk about privilege is that there is a society-wide context where being white or whatever gives you privilege.
It's not like America has a white police force that discriminates against black people and a black police force that discriminates against white people. We have one police force and it's a white one (in the sense that it tends to be run for the sake of white people and embodies white values).
It's not like we have a white tech sector that discriminates against black people and a black tech sector that discriminates against white people. We have one tech sector that is run (disproportionately) for the sake of white people and embodies white values.
Does that mean that every white person has it good? No. But being white does have privilege associated with it. You're part of the dominant group and have an easier time navigating the social world because of that.
That being said, it's possible to be white and also mentally ill (for example). That doesn't mean you don't have white privilege, it just means you lack the privilege of people without mental illness. Saying someone is privileged doesn't mean that their life is good, it just means it's better than it would otherwise be.