My read here is that you're implying that if an attacker has access to, for example, weight data, they can invariably find a way to exploit it.
If that's a correct assumption, I think you're playing an unwinnable game, since attackers always have indirect access through inference of the model. It feels like locking down weights/training data/etc is the ai version of security through obfuscation.
Just my 2c, for what it's worth