Everyone is giving agents their own sandboxes these days, so that part isn't new. But I wanted to share the "why" and what we learned from building them. We first built them the way you build anything else in the cloud, but agent sessions don't behave like cloud jobs and a lot of cloud habits were wrong for them.
A few examples:
1. We first built a microservice that was responsible for managing each agent session, and thought we could scale it horizontally. That doesn't work, not if you want to give your agents a real file system and tools that need to be installed on that machine. We later gave each machine its own sandbox.
2. Our agents power our UI, but also live in Slack, Github threads and wait for people to reply. Our sandboxes had a 1-minute timeout and a cold-start. So once a sandbox died, it took anywhere from 10 to 30 seconds to spawn, because of which the users had to wait longer before they received a response from our AI on the UI or in the threads. Now, we maintain warm pools of sandboxes which can be acquired by a session in milliseconds and each sandbox is kept alive for 5 minutes after each reply.
The post is fairly high level, but if you want more detail on any part of it, I'm happy to talk about it here.