People think "RSA is easy", because someone gave them a lecture of a simplified/wrong/insecure version of RSA. Pretty much all "simple introductions to RSA" you can find out there are wrong.
The truth is: RSA isn't that simple. If you want to have RSA, and want to have it secure, there's a whole bunch of things to consider. But RSA looks simple.
So lots of people go ahead and implement their RSA. And then you end up with, hey, I can break almost a third of the top 100 webpage's RSA implementations. (I'm not kidding, I did that -> https://robotattack.org/ .)
I think the fact that crystals-kyber is obviously not that simple may actually protect us. Because hopefully most peopple will end up using some hopefully well audited optimized free implementation, because they won't even have the idea that they could do this on their own.