The real problem with AI, as I see it, is that we are not ready to give the average person the capabilities they will soon have access to. The knowledge necessary to build nuclear weapons has been readily available for decades, but the constraints of the physical world put practical limits on who can do what. We are also able to limit who can do what by requiring formal education and licenses for all kinds of professions. This is no longer going to be strictly true; or at least not in the same way. If we impose no restrictions, literally everyone on the planet will eventually be able to do anything, provided they have the resources.

The fact is that AIs are more likely to be accomplices than masterminds, at least for the foreseeable future. What we are afraid of is that they will end up being just as terrible and flawed as we are, but it's more likely that they will do what they do today: what we ask them to do. The greater risk is therefore from malicious internal users rather than from the technology itself.

Perhaps more to the point, there is no effective way to limit the output of an LLM or place restraints on how they work. I think it's foolish to even attempt that -- they just don't work that way. The better way to regulate this kind of technology is to focus on the people. Some data classification and licensing program seems a lot easier to implement than the road we currently seem to be going down; which is either no regulation or insanely restrictive regulation.