"Sexist" does feel like too strong a word. "Gendered" perhaps more appropriate.

"Guys" is certainly gendered, so I understand folks feeling like "you guys" is as well.