Also "agency" isn't a requirement for almost any AI doom scenario I've heard of. Goals and the formation of instrumental goals is all it takes.
In order to have goals you need to have a model of the world. LLMs do not have a model of the world.
Your goal then is changing this world somehow so you need actuators.
So you need a model and actuators.
All we have is a stream of words appended in a probabilistic manner.
Current "hyperintelligent AI" fears are exactly like the "grey goo" fear when nanotechnology was a buzzword in the early 2000s. We can all agree that if you take the fundamental vague concept "AI" or "nanotechnology" and pile on a couple of wheelbarrows full of hypotheticals, you get to something scary. That doesn't mean it has any relevance for the universe we currently exist in.
People anticipating future risks and discussing how and when to detect and mitigate them is actually exactly what we should be doing.