AI can be as smart or smarter than most humans in many ways, but I think it's a very real possibility that its development path won't render the human mind useless. They key difference between AI and humans is AI has the power to iterate and learn from its mistakes much faster than humans without fatigue. The methods of which it learns are created by humans. To assume the creation of AI with "a positive feedback loop that goes nobody knows how far" without humans first understanding how seems more of a belief in magic to me.
"I don't think, and a lot of smart people don't think, this is true."
When it comes to predictions, smart people can be wrong. I could be wrong or they could be wrong, and they may be smarter than me, but I'm smart enough to know this is true.