I like John and his body of work, but this and the general slew of posts with very little actual knowledge or expertise making grandiose / prophetic assertions are the soothsayings of our age - people convinced that they've cracked "it", but without any factual reasoning as to why.

They raise philosophical assertions disguised as logical / scientific assertions.

> “Better.”

> That is a concept that cannot be made via cold calculation. A universe boiled down to particles and numbers cannot care whether atoms arranged as a suffering plague victim are “better” than those arranged as a healthy child, or a plant, or a cloud of gas.

It can, John. The engineers teach machines what's good and bad. That's the first step in creating silicon intelligence.

> It is impossible to understand pain without having felt it yourself; it does not convey as an abstract value. I think the machine needs to tangibly interact with human suffering in the same way the Roomba’s wheels need to physically touch the floor.

Pain is nature's deterrence mechanism, bred into us through evolution. Deterrence mechanisms can take many shapes.

> This leads us to the ultimate cosmic joke: This task requires answering for this machine questions that we ourselves have never been able to answer. Let’s say I’m right and this system won’t exist until we can teach it emotion, and the capacity to suffer. How long until it asks the simple question, “What is our goal, long-term?” Like, as a civilization, what are we trying to do? What are hoping the superintelligence will assist us in achieving?

I hate being disingenuous and make a flippant argument here, but reading this from start to finish makes me ask a bad faith question - your assertion is that we won't have self driving cars because we can't tell the robots what to do with the whole civilisation right away? There's a whole spectrum of amazing (or terrible) possibilities between the extremes, my guy.