The major problem with this "people of color" designation is that it sweeps specific discrimination against African Americans* under the rug. It is certainly possible to have 100% employees "of color" while consciously excluding African Americans, which I have witnessed at private companies (blue collar jobs mostly) in California (where I live). The result is that affirmative action for African Americans, which has been replaced with affirmative action for women and non-white men, does way more for non-African Americans than the originally intended recipients (collectively). Even black African/Caribbean immigrants are used for diversity accounting-- see the NYTimes article from 2004, "Top Colleges Take More Blacks, but Which Ones?".

Please note this is not a slight against women, immigrants, or anyone else. They obviously have valid discrimination claims. But if you work in a community where less than 1% of African-American students are scoring at grade level in mathematics, and your company claims it is "diverse" yet hires no local African-Americans, there is still a serious problem in your community that should be addressed. (It's OK if you don't care, as long as you don't pretend otherwise). So please do not conflate hiring "people of color" with hiring any specific group.

* By "African-American" here I mean descendants of slaves who lived in the USA, and are thus severely disadvantaged because of it, having actual wealth stolen without any legal means available for recovery. I do understand that "African-American" is commonly used to mean any black U.S. resident.