It's quite rare for that to be the case, at least in the English speaking side of the Vtubing world. I can really only think of one example.

That used to be the case, but dudes through a voice changer are quite rare now.

This is a very tired insult/joke outsiders throw around about vtubing (though I don't get the impression that you had that intention). It's somewhat ridiculous, vtubers stream very often and voice changers tend to be pretty obvious. There are some men who use female avatars, but it isn't common.

It's also kinda misogynistic, plenty of women enjoy anime culture. Many prominent artists for vtuber models are also women (and occasionally vtubers themselves). To the level that calling the model artist "mama" and the model rigger "papa" is a common trope.