Seems like so many people have been pissed about this line. It seems obvious that he didn't mean "women should not have the right to vote" though... just that it makes libertarian ideals harder to achieve, because women tend to favor more government.
Hence his main argument: The US majority will probably never be libertarian, so libertarians must try to go beyond politics if they ever want to achieve their goals.