> Indeed, focusing on this particular threat might exacerbate the more likely risks. The history of technology to date suggests that the greatest risks come not from technology itself, but from the people who control the technology using it to accumulate power and wealth.

This is where you lose me. AGI (not LLMs, but intelligence smarter than us) is qualitatively different from anything else we've ever made. The one example we have of this in history is the development of human intelligence. It did not work out well for other species, and now we own the Earth. This isn't a perfect analogy! There are some reasons why AGI may be better (we'll be actively trying to align it to human interests) but also why it may be worse (there will be pressure to make it more powerful and take off guardrails).

There may be good reasons for why we'll be able to control AGIs, but "history of technology suggests..." is not one of them.

And I'm not arguing that all the "human misuse" problems are not important! They're hugely important! A likely outcome, the way we're going, is that a bunch of people get really rich and everyone else becomes super poor and possibly starves, and wars break out. I just think that, after that, AGIs also end up in charge of the planet and from there we either die or we have no say in how things go from that point on. I don't object to working on the former problem, except to the degree that the solution makes the latter problem worse (like, "open source all the models", which is likely to happen at some point anyway).