Normal food has the advantage that we already know it has no disastrous ecological consequences, by virtue of the fact that is has coexisted in its environmental niche for a while. The more food is altered, the less sure we become about that.
As Taleb has pointed out, our standards for danger should match the risk/reward involved. (E.g., we're ok with fast-tracking unproven cancer treatments because the alternative is death. Likewise, we have high standards for a common cold treatment because the alternative is inconvenience.) Unfortunately, we have no backup plan for the planet, so we must be extra careful with what we introduce to it.
To me, saying we don't need more stringent standards for GMOs is like saying we can definitely burn $X billion more oil reserves and not worry about global warming.