This is a human problem too. It's not limited to AI. Think about the Two-man rule in nuclear weapon activation. We can't trust one neural-net to receive and confirm the "launch" prompt as valid, so we use two people to increase guarantees. It's called SoD , segregation of duties, by risk management people.
Some architectural changes to how LLM transformation works could include:
- Create a separate tokenizer for system prompts, so that system embeddings are "colored" differently in the model. This will, however, complicate training and bloat the model into uncharted computing territory.
- Create a separate set of pre and post-prompt AI sanitizers that DO NOT use user input or generated output as part of its instruction. Text in and out of LLM is always tainted, so it's a goal to avoid it as input as much as possible.
Simple classifiers can be used for sanitation, but they tend to be "post facto": given a prompt injection scheme comes to light or an prompt injection incident is reported, train on it. More sophisticated intention analyzers, backed by deep classifiers that are uncertainty-aware, and beefed up by LLM generational tools pretrained on synthetic injection schemes, could probably detect ill-intention accurately in the same manner sentiment analysis can pick up on very subtle user queues.
The issue is that classifiers would still be dealing with marshaled code+user input. I believe the better option for intention classifier inputs is to use the model processing data (ie. a "generation log") as input to the classifier, similar to how the ventromedial prefrontal cortex and amygdala connect, acting as behavior moderation neural nets in us humans. This would typically be done by adding specialized multi-head attention focus areas in the GPT architecture without the need for separate classifiers, just basic training about what is good and bad for the AI, but then we're back at the original problem of dealing with the input text directly.