That argument has been used to defend all sorts of terms, some worse than others. The example you choose "guys" as gender neutral is also common in US english. But being common does not mean it is correct (racial and homophobic slurs were also common for a long time).
- Group of women === "girls"
- Group of men === "guys"
- Group of women with one guy === "guys"
- Group of men with one girl === "guys"
The "guys" as gender neutral is clearly the result of patriarical society. But does that make the word _harmful_? Hard to say. I lean towards "no". But I'm not in the adversely selected group.