You're missing a very critical detail.
Normally, for a trait to grow in a population, you need a fitness advantage. Basically, if you average a little over 2 offspring that successfully breed, that trait should grow to fixation.
In the case of exactly 2 reproducing offspring, allele count is static. Homozygous parents will provide one allele to each offspring, so 2 copies in one generation, and 2 copies in the next. Heterzygotes have a 50% chance, so one offspring is expected to have the allele. 1 copy in the parent generation, 1 copy in the next.
Gene drive changes the completely, as the construct will copy itself to the homologous chromosome. You'll go from 2 to 4 or 1 to 2 copies from the previous example. The growth is exponential. There is nothing like that in nature.
Thus, even if a moderate fitness disadvantage is introduced, the allele can still be driven to fixation quite easily. Even with an expected offspring of 1.5, you can still go from 1 to 1.5 copies of the allele over each generation.
The other issue is that you can introduce a trait for e.g. chemical susceptibility. It offers no fitness disadvantage until that chemical is introduced into the environment.