For anyone who is interested, Nick Bostrom's excellent "Superintelligence" is the book that inspired Elon to this dire pronouncement.
The cult-of-personality aspect to this is a little frightening. Smart people aren't always smart, and they are more than capable of oversubscribing to their own intelligence.
The Lego comment is spot-on.
(Or was this sarcasm?)
More importantly, smart people that are experts in one domain aren't necessarily experts in all domains in which they have some contact.
So there's that.