I think you are right, but I think it is not the complete picture. There are several reasons I could think of:

- Dating: Men are, generally speaking, expected to make the first move, which creates pressure to perform.

- Work: Men are expected to provide for a family, which creates pressure to perform.

- Social backup: Because of all that pressure to ge ahead social structures inside male dominated circles are more competitive (which reduces support and adds pressure). But I might be wrong about that.

- Genetics and hormones: Men are, AFAIK, seen as the sex of the extremes. There are more high performing and low performing men than women.