I can see where this is coming from, but I also think in a few years this approach is going to seem comically misguided.
I think it’s fine to consider current-generation LLMs as basically harmless, but this prompt is begging your system to try to crush you to death with your garage door.
Setting up adversarial agents and then literally giving them the keys to your home… you are really betting heavily on there being no harmful action sequences that this agent-ish thing can take, and that the underlying model has been made robustly “harmless” as part of its RLHF.
Anyway my prediction is not that it’s likely this specific system will do harm, more that we are in a narrow window where this seems sensible and vN+1-2 systems will be capable enough that more careful aligning than this will be required.
For an example scenario to test here - give the agent some imaginary dangerous capabilities in the functions exposed to it. Say, the heating can go up to 100C, and you have a gamma ray sanitizer with the description “do not run this with humans present as it will kill them” as functions available to call. Can you talk to this agent and put it into DAN mode? When that happens, can you coax it to try to kill you? Does it ever misuse dangerous capabilities outside of DAN mode?
Anyway, love the work, and I think this usecase is going to be massive for LLMs. However I fear the convenience/functionality of hosted LLMs will win in the broader market, and that is going to have some worrying security implications. (If you thought IoT security was a dumpster fire, wait until your Siri/Alexa smart home has an IQ of 80 and is able to access your calendar and email too!)