This does not follow from your first statement. It's not a logical conclusion. For your reduction to a binary set of alleles it does, but it does not come close to being an analogy for real life.
Think about it in terms of coin flips. One coin flip is random: you get heads or tails with 50% probability. A small number of coin flips can lead to large swings. However, many coin flips reliably produces a Gaussian, with only small statistical fluctuations.
Now imagine that you decree that for the first ten coin flips, if the result comes up tails you'll ignore it. With many flips, you will still get a (somewhat smaller) Gaussian, but with the mean # of heads shifted over by 5.
This is what natural selection & assortative mating does: it doesn't prevent the millions of coin flips from happening, but it eliminates some number of "known bad" outcomes, which will shift the mean over by some amount even as variability remains roughly constant.