That can be true, and I think it's particularly common with imposed, rather than adopted identities. In fact, people having particular ideas about you that aren't necessarily true is the reason a lot of imposed identities exist in the first place. There's not a lot you can really do about it. I suppose you can try very hard to not identify with anything, but that's not always an option, and even when it is it would be a very lonely existence.
> And in general, I think people who make $IDENTITY the main feature of their personality will have difficulty outside of that one social group.
Another way of saying that is that people who aren't well rounded will have trouble in a lot of social situations. I agree with that, but I don't think it's a problem related to having identities so much as it is someone restricting themselves to a single focal identity.