The architecture of the current models where learning is a separate and very expensive process make runaway self improvement seem like something that will require a bunch of breakthroughs in order to happen.
The current LLM approach makes huge assumptions, including that training only on text prediction is enough to simulate true intelligence. That may or may not be a valid assumption, but it could be enough for the LLM to make one seemingly small change that ends up running away from us faster than we would realize.
More importantly, I think we agree a sudden run away is possible, we don't have safeguards in place, and the risks are extremely high should that happen.