Today, 50% of people who get degrees in law, medicine, or accounting are women (up from basically nil in the middle of the 20th century). Is there anything more "masculine" about looking through a hex dump to debug a program than there is about poring through Excel spreadsheets to tie a deduction to an expenditure? It's all dry, analytical detail work at the end of the day. If anything, going by the usual stereotypes, programming is more suited to women--it requires a lot of patience and involves a creative aspect.
Is it a matter of mathematical ability? Possibly, but that doesn't explain the disparities you see. The differential between men and women among people with perfect SAT Math scores is 65:35. At the 700 level, its 60:40. Even if math skills are the most important thing, they don't explain the gender disparity among programmers (though it might explain the gender disparity among Fields medalists). If ~40% of all engineers were women, we probably wouldn't be having this conversation.