AI model rules will be as successful as any other prohibition, where outlaws will act with defacto impunity, while good people who commit sins of omission will be made arbitrary examples of. I'm sure there's a name for the dynamic, where policing rules of any kind are mainly enforced against people who generally abide by them, while simulataneously giving a huge arbitrage advantage to people who ignore them or are just outlaws.

There is another problem that doesn't have any good solutions yet that will be a huge part of AI governance, and that's software attestation (direct anonymous attestation). The basic problem is how does a program assert that it is an authentic instance of itself. We've been trying to solve it in security for apps, authenticators, and DRM for decades, and the solutions all seem fine until it's worth it to someone break it. I think it's probably just a poorly formed problem statement that defines itself in impossible terms, but when they can't govern AI models, they're going to try to govern which AI's can access what data and systems, and we're back to solving the same old cryptographic problems we've been working on for decades.