It's the same in Japan (15yrs in Japan)
That is not to say you can't find people of both sexes into anime. But, when I've asked at parties, unless it's specifically a nerd party, 19 out of 20 times people say they don't watch.
Back in the USA, at least at my age, it's frustrating being on the dating sites, in particular OkCupid, which has the questions "Would you date a comic book geek?". If I filter on that question it goes from thousands of people to just 3 in a 50 mile radius of SF. This dispite the fact that many famous people, especially famous movie and TV series directors are comic book geeks. Maybe the wording "geek" is enough to influence the result.
Another interesting one is "Would you enjoy a night playing video games". Just 2 women. This dispite the fact that the gaming industry says 45% of gamers are women. Either that stat is BS or else all the gaming women are in relationships.
Another one I find even more alarming and makes it feel like something fishy is going on "Do you like to wear a costume for Halloween?", just 3 women. But go to any Halloween party and it's fully mixed and seeing the Spirit costume shops all over the place someone is buying these costumes. Again one guess is that the people fun enough to wear a costume are not on the site.