My core claim is that Bostrom doesn't know his arse from his elbow. And a professorship in philosophy at Oxford is not, in itself, a great support for his authority on technical matters, or on the behavior of intelligent species yet to exist. That is, in fact, a topic on which no one speaks with authority.

I have nothing against science fiction, but I object to any fiction that disguises itself as non-fiction, as Bostrom's often does. Nor do I think that the impossibility of predicting the future of AI is, in itself, a reason for undue caution.

Bostrom is performing a sort of Pascalian blackmail by claiming that the slight chance of utter destruction merits a great deal of concern. In fact, he is no different from a long line of doomsday prophets who have evoked fictional and supposedly superior beings, ranging from deities to aliens, in order to control others. The prophet who awakens fear puts himself in a position of power, and that's what Bostrom is doing.

Regardless of Bostrom's motives, we as humans face an infinite number of possible dangers that threaten total destruction. These range from the angry Judeo-Christian god to the gray goo of nano-technology to Peak Oil or Malthusian demographics. In each case, the burden of proof that we should be concerned is on the doomsday prophet -- we should not default to concern as some sort of reasonable middle ground, because if we do, then we will worry uselessly and without pause. There is not enough room in one mind to worry about all the gods, aliens, golems and unforeseen natural disasters that might destroy us. And you and I both know, most if not all of those threats have turned out to be nonsense -- a waste of time.

I do not believe that Bostrom carries the burden of proof well. His notion of superintelligence is based on a recursively self-improving AI that betters itself infinitely. Most technological advances follow S-curves, moving slow, fast and slow. Bostrom does not seem to grasp that, and cites very little evidence of technological change to back up his predictions about AI. He should be, first and foremost, a technological historian. But he contents himself with baseless speculation.

We are in danger, reading Bostrom or Russell or Good, of falling into language traps, by attempting to reason about objects that do not exist outside the noun we have applied to them. The risk is that we accept the very premise in order to fiddle with the details. But the very premise, in this case, is in doubt.