If I met a tiny petite girl in the woods, while I was hiking, and if she wanted to stick around with me, I would also be wary of her. I would think that she's maybe part of a some elaborate robbery scheme or something. Stranger danger and all that.
And if a man, or anyone, finds themselves in a position of power, and they decide to abuse that power, that is on them. There is no social norm that teaches men to take advantage of women whenever they feel like it.
Ultimately this is a worthless discussion because you cannot legislate evil away. Some people will always behave badly and no amount of preaching about the patriarchy will change that.