Without any further information, given what we know about sperm production (it's a physical process for which we're measuring time), the expected distribution is not normal (probably exponential or something). The claim that 'variance goes both ways' is wrong. Variance can only go one way. It can go one way mostly, or it can go bothways, or it can take a U shape. It all depends on the distribution of the events.
If we take it to be a simple exponential distribution with mean 66 days, then lambda = 1/66. and variance is 4356 or a standard deviation of 66 days as well.
In that case, by month 4, about 83% of men would be expected to have sperm in their semen.
To delve deeper, what you're missing here is the idea of skewness and kurtosis. These are the 'higher moments' of the probability distribution function (natural extensions of mean, variance, etc). In particular, the skewness is where you're getting hung up. You assume in your first sentence that the skewness of every pdf is zero (i.e., it 'goes both ways'). That is not generally the case.
In particular, the skewness of the 'time to first sperm' distribution must be very positive because it can never be below zero.