Do you also get mad when people use the word "guys" to refer to more than one person of either sex?
"come on now ladies lets move the tour along" "Who you calling 'lady'!?"
Guys really is just a historically ingrained shorthand. I don't think I've ever heard it used in this sort of context to cause harm. Using "Ladies" in this way, on the other hand, could cause yourself harm.
As much as anything, I've heard many women refer to mixed gender groups as "guys" or "dudes". A lot of this "definition creep" is actually due to women themselves. Probably this is because the male identity is viewed as a source of power, and women seek this out and aim to identify with it.
In other words, I don't think the use of "guys" generically is being driven by sexist men. I think it's being driven by blurring of gender roles and definitions, and adoption of these generic terms by women.