We have this idea that once we get to AGI we stop research, or that’s when we put a pause and think about how we are going to use this technology responsibility.
I think the issue is, if the computer is more intelligent and capable than 10% of people, then it will already vastly change our world (we’re probably close to this or surpassed it)
Can you prove that any other person you know is generally intelligent, and not just a LLM spewing statistically relevant words ?
You assume they are intelligent based on their actions and abilities, so I guess we will do the same with AI.
Fake in this context is not any more dangerous than a con man (wants to harm, but it’s not competent), which is already pretty dangerous, but not as dangerous as somebody who is capable and interested in harming.
It's fairly intuitive that LLMs are already super-intelligences, and if they don't stay "alive" continously, is just a matter of design. We could be one simple - as in bash - "while (null) do, done" of having a thing hardly distinguisable from AGI (even if it's not AGI, whatever you can get out of that "I" letter to precisely define it as being a real one or not).
I'm sure most super-powers are already peaking a sight into the most confidential information inside OpenAI, DeepMind, Alibaba, Meta, whoever has these things running and keeping the details hidden from the general population.
For I, I'd do it, just in case some wacko start to run some of the things inside a while loop, giving some kind of ability to retrain itself and/or access to the real world (Internet access or worst, ability to control drones to physically interact with the non-digital environments).