Very interesting. I've heard stories of the GMO spreading to another farmers field, causing some problems. But that field is mostly devoid of insects, so it's easy for the GMO to live there.
I still think that a fear of "irreversible system-wide effects with unknown downsides" will persist, despite the good sense of your argument.
Also, as a technologist, I know that we move from solving easier problems to more complex ones. Who is to say that GMO tech isn't just beginning in farmers' fields, and that flush with success in that space, all those engineers, technology and know-how will happily move on to the Next Big Problem. This might be in wild-plant space.