Just to clarify, I'm definitely not saying "neuron simulation" is required in any way. I'm just asking, how can we be very close to "solving" a significant part of the most complex brains, yet miles away from solving the simplest brains?
You should be able to answer that question (or a steelmanned version of it), not just ridicule strawmen.
Empirically, it does not seem necessary to understand one version of a thing to produce a superior version. Why should the remaining unsolved cognitive tasks break that pattern?
The immediate threat is that humans will use the leverage that LLMs give to replace and to influence other humans, in other words to gain power over us.
Whether this is AGI or not is beside the point.
As for the influencing part, what specific actions to gain power over us can be achieved now with LLMs, that could not be achieved before using a few tens of thousands of paid humans?
It's being able to do it without having to employ the tens of thousands of humans that makes it different. With the LLM you are able to react much faster and pay fewer people more money.