Let me save you 20 minutes of your life: This guy's lead argument is that the smart people who advise caution in creating super-intelligence look weird. He's quite proud of this argument.
*"But the outside view tells you something different. ... Even though their arguments are irrefutable, everything in your experience tells you you're dealing with a cult. Of course, they have a brilliant argument for why you should ignore those instincts, but that's the inside view talking."
One should note that the AI people thought AI would look completely different than an LLM does (they thought it'd be an agent using a manually coded knowledge graph) but are still applying all their same arguments to this very different thing.
AI super intelligence arguments have the same logic: the data ends so the interpolation now goes to infinity. This is such an obvious counter and these people know this math.
Extrapolation not interpolation, proceed with downvotes.