I think you missed the point. It's not that "people give up a little when others tell them that they are bad..." The effect is self-afflicted, you just have to think that there is a negative stereotype that affects you, and your performance will suffer. Moreover, the effect is actually stronger if you think that you are pretty good at whatever the stereotype says you should be bad at.

The point is that, regardless of whether the stereotype is true or not, this is a scientifically established psychological reaction that causes people to perform below their capacity, and it only affects those that are from a stereotyped group. This means that, in your example above, women will constantly be at a disadvantage compared to men in math performance.

Does it matter? If you were a member of such a group, I'd wager you would think it unfair, and doubly so since you are already trying to break a negative stereotype.