Putting it inside docker is probably fine for most use cases but it's generally not considered to be a safe sandbox AFAIK. A docker container shares kernel with the host OS which widens the attack surface.
If you want your agent to pull untrusted code from the internet and go wild while you're doing other stuff it might not be a good choice.