The idea of language contact reducing grammatical complexity isn't absurd, especially if you approach it from a linguist's perspective, and not a lay perspective. The misconception I think most people have is that the goal is to ask if English or Mandarin is more complex than the other--and that's a fool's errand. But it's not what's being observed. Instead, compare English to other Indo-European languages, especially its closest Germanic relatives. English is lacking in the Indo-European penchant for inflection, and you can pretty narrowly pinpoint the moment it happened: the influx of first the Danelaw and the Normans and the concomitant shift from Old English to Middle English.

If you then look at other languages, you can find other cases where individual languages seem unusually simple compared to their relatives. Swahili, the East African trade language, is rather simple compared to its other Bantu brethren. Malay is simpler than its nearest Austronesian relatives. Even Old Chinese does seem like it had a similar grammatical crash to English (compare it to the other Sino-Tibetan languages), just 3000-4000 years ago instead of 1000 years ago. You can also find cases like the Kartvelian languages, where the comparatively more isolated regions seem to produce somewhat more complex languages than the somewhat more easily accessible regions.

There's also a believable putative mechanism. It is a common observation that adult language learners tend to have a harder time learning languages than children. It's not hard to imagine that a language being learned by a lot of adults ends up having those new language speakers shave off some of the harder bits of the language, and those adults then passing on this shaved-off language to their kids (who are of course unaware that they're learning a simpler language).