But to me (not a native speaker), "person of color" is vague and misleading. I didn't even automatically know it's about the person's skin color. First I thought it means a flamboyant, over-the-top person.

If I think about colors in the scientific sense (black: lack of light, white: mixture of lights from all colors), the "person of color" expression would include everyone except black people, but how it is used, it includes everyone except white people (wherever we draw this "white-enough" line, depending on the context).

Why not just use "non-white person"?