Like you, I have no problem eating GMO crops but as you say, there are potential knock-on effects from a systems perspective. Similar to the financial system, if the sole pursuit is yield, systemic problems arise.

The main issues I have with GMO is the patenting of crops, specifically those that are introduced to third world countries which are typically sterile, causing the farmers to have an ongoing dependence/obligation to the patent holders. Probably an even greater problem is crop resilience - yes, many crops will give far greater yield when given ideal conditions, but by reducing diversity within the same species makes them more vulnerable to specific diseases. In nature, there is always some % of a crop that's resistant to a disease, that diversity is removed when all the crops are essentially identical.