No, that's not really the case. It doesn't matter if something is done on the basis of race or sex. If the end result hurts those groups disproportionately it's racist/sexist.

I guess this just may be a difference in definitions, but I think when most people are talking about racism/sexism on a macro level, they are using this definition.

You're right, the broader issue is worker's rights, but it's definitely worth pointing out that among workers with the least rights, minorities are overrepresented at Amazon.