Again, I don't want to take sides, as this "anti GM"/"pro GM" seems to be ridiculously religious matter in the USA, but since you asked, I guess I can answer.
The difference is, basically, that random mutations happen all the time since life on Earth has begun, they are relatively subtle, and, last, but not the least — unavoidable whether you like the fact that they happen or not. They are pretty close to definition of "normal", because that's just how carbon-based lifeforms (or, probably, any theoretically possible lifeforms at all) work. The result of any individual mutation might be not pleasant: child might be born defective or ugly, but that's… well, "natural". People tend to avoid unusual, so even though some ugly chicken with two peckers and one eye might be completely healthy food — many will think it's disgusting and won't eat it. And, as I said, random mutations cause relatively subtle changes in phenotype, so it would take many, many generations for such chickens to become normal even if we selectively breed them to have two peckers. Even single such mutation is something pretty unlikely without strong external influence, like high radiation or some chemicals. And if specie changes a lot during multiple generations — it becomes normal. If they survived through so many generations, and we still eat them: it somehow "works", one might think.
Genetic engineering, on the other hand, didn't happen for a couple of billions years until last century, is very much avoidable and immensely powerful. Consider making a pig glow through selective breading. Seems pretty unlikely, yeah? Well, it's simple now, if you tinker some with genome of a pig and a jellyfish. And that's just an illustration: possibilities of what you can produce are nearly unlimited. I don't want to quote Spiderman's uncle here, but I guess you get the point. One might not even actually think glowing pig is dangerous, but it's certainly suspicious. Which is more than sufficient reason to beware of something or to want to avoid it.
The best thing about living nature is that something, that can seem completely safe and uninteresting at first, might be a cause for something much bigger than it seemed. Like, you know, bringing a couple of rabbits to Australia[1].
As a side note, you should probably remember when arguing with those "anti GM" friends of yours, that thinking of something as dangerous isn't the only reason why somebody might not like something. Say, they might happen to not like green wallpapers — it doesn't necessarily mean they think green wallpapers are dangerous. They just don't want them, maybe for religious reasons or whatever. So if everyone suddenly starts producing only green wallpapers because it's simply cheaper, and any other colors become rare, ridiculously overpriced and are treated by sellers as a luxury product — they can become "anti-grean-wallpaper" very easily. In attempt to bring balance to the force, if nothing more. Add to that that not everyone is perfectly rational — which is kinda ok as well — and there can be infinite number of reasons why somebody might be anti-something.
[1] - https://en.wikipedia.org/wiki/Rabbits_in_Australia