Do many "otherwise smart" people actually believe "superhuman machine intelligence is prima facie ridiculous"? I'd like to see some citations :-). I think smart people tend to have much more nuanced views.
I don't know how "otherwise smart" I am, but I wonder how we would be able to tell that a machine intelligence was "superhuman" as opposed to "buggy".
For example, suppose we build a super-AI and ask it, "Is Shinichi Mochizuki's proof of the ABC conjecture correct" [1]. What would we do if it said "yes"?
(Of course, if "superhuman" just means "able to do things humans already know how to do and verify, but lots faster", then we're already there).
[1] http://www.newscientist.com/article/dn26753-mathematicians-a...
Yeah, that would work :)
Maybe the question I should have asked, is:
What if we ask a super-AI for a proof of the ABC conjecture, and the result is something too complicated for humans to verify?
My point, if I have one, is that when I read about "superhuman machine intelligence", sometime people seem to mean "capable of knowledge that humans couldn't figure out on their own but that humans can understand once they see it"; and sometimes they seem to mean "capable of knowledge that is beyond human capacity to even verify".
I think development machine intelligence of the first kind is extremely likely, but I'm more skeptical about the second kind.
The Strong AI hypothesis assumes a mechanistic universe, if not necessarily a materialistic one, and I think that condition is false.