The difference is definitely genetic. Colorblindness is generally caused by recessive genes that are part of the X chromosome. That means that the proportion of men who are colorblind is basically equal to the proportion of X chromosomes with one of these recessive genes. Women have two sets, so they need the gene on both chromosomes in order to exhibit colorblindness. This means that their proportion of colorblindness is roughly the square of that of men (actually less because there are different types of colorblindness).