But precisely how does it accelerate if its instrument in the real world to do harm isn't already trained in the domain of doing so?

There's a huge difference between woodworking and watching an instructional video about woodworking unless it is a very well crafted video about woodworking. AI slop to this day does not fit the bill. just try watching one of the AI slop videos about AI to see what I mean.

There was a tweet about someone going to visit a doomer Meetup and it was taken as an axiom that AI will kill us all in the next 10 years. At one point they asked people to raise their hands if they were skeptical about it. Those who did were told to leave the room and the doomer revival meeting continued apace.

That's not science or engineering, that's pure religion. So if your starting axiom is that AI is going to kill us all in 10 years, we really don't have much to talk about because I just disagree and would put the probability of that well below 1% (nozero of course if only to avoid divide by zero errors).