This is so cool! I have been wanting to see something like this for a few years now. I tried making a demo of something similar (but much more primitive) in Unity back in 2021, but small transformers weren't good enough at the time.

Is there any way to protect against prompt injection here? Looking at the architecture I am thinking it would be possible for users to tell an agent what to do directly by tricking them.

This isn't really a criticism, I think it's actually a cool feature. It might be a fun premise of a game where you know you are in a simulation and can manipulate the NPCs around you.