I just wanna say, I really admire the irony of this sentence. Maybe once we invent superintelligence it will be able to think enough to tell us how badly we needed it before we had it.
> The urgency is the highest, especially ever since simil-intelligence (LLMs) was implemented.
It isn't. If we stopped all work on LLMs and AI we'd be more than OK. There's no imminent danger or threat.
> To research Intelligence "safely" you simply do not give it agency (you do not let it control computers), for example.
Something can't both be smarter than us and completely controllable by us. The whole point of inventing superintelligence is so it can do things we can't. Humans can already break out of sandboxes, invent bioweapons, manipulate each other to invent bioweapons, etc. What makes you think superintelligence can't or won't do these things?
> To have "safety" in general, you cannot empower morons - humanity is doing the opposite. We cannot rest with incomplete LLMs, nor we can with this dumb, intrinsically dangerous humanity.
Hey, you're talking to a guy who's got a list of tech to ban: nukes, social media, all forms of video news, etc. What does Taylor Swift say, "I've got a blank space, baby, and I'll write your name (Claude)".
> What does "safely" mean? Getting stuck with this infra-humanity?! That is the radically unsafe thing.
You literally just ban it and turn it off. People are always like "you can't ban it bawwwwwww!" but we ban tech all the time. Try buying a bunch of fertilizer, try buying equipment to research gain of function, try buying anti-aircraft guns, try building fuselages to enrich uranium. LLM research requires extraordinary amounts of hardware, electricity, data, funding, and all the things upstream of that (cooling, shipping, space, financing, etc.) The effort you'd have to go through to do this clandestinely, illegally, while also finding researchers willing to take the risk is all prohibitive. There's a list of all nuclear scientists in the entire world. They have to go to school somewhere. Same for AI researchers. If, say, half a dozen of them suddenly disappear, and a random town in Wyoming suddenly starts making millions of requests per second across the whole internet, guzzling electricity, producing a lot of heat, or ordering a shitload of insulation and ducting, we'll know!