MicroVMs: Run isolated sandboxes with full lifecycle control

Thanks for sharing these!

Why isn't libkrun good enough for hosted stuff? I use it as a podman backend in a microservice architecture.

Setting up your own is not that hard and if you bought some compute before the Altman squeeze, very cheap.

What people aren't getting with `firecracker` is utilization. Don't get me wrong, `firecracker` is great software and it's what I'm using for lightweight virtualization, but workloads are really bursty over really short periods of time now, even with the snapshot and restore that you can get if you're willing to hack on `firecracker` substantially, you hit walls where it's like, this is too much against the grain, this thing wasn't designed to bounce from 1 core to 32 to 8 to 16 to 4 to 32 to 1 seamlessly, and that's what it takes to get extreme utilization even with extremely good ML on the prediction.

I am quite sure I'm not the only person working on post-firecracker KVM.

We’re adopting agent-sandbox (https://agent-sandbox.sigs.k8s.io/) as we already run most if our stuff in kubernetes and it’s been working very well, the only downside being it’s a moving target as it’s still essentially in development.

Agreed.

Most of the startups are just wrappers around AWS and significantly more expensive.

Agents need sandboxes that are cheaper so that they can run thousands

I feel that AWS, GCP and all the other cloud providers can provide this natively.

But still it would be nice to self host.

The best part of self hosting is that you own it as well, no rug pulls from the laundry list of reselling providers that could go away at anytime.

It would be nice to have a one click sandbox agent on a self hosted instance that is, free, fast (can pay a bit more for more intensive operations) and that is open source.

AWS AgentCore runtime has been around for about a year: https://docs.aws.amazon.com/bedrock-agentcore/latest/devguid... (spoiler, it's the same underlying technology as the Lambda MicroVMs).

Major Sandbox providers (e.g. Modal) run on non-hyperscaler bare metal not AWS and so don't need to markup on AWS's markup. Thus, prices are comparable or better than AWS.

Checkout https://shellbox.dev for exactly thisnusecase: boxes can be stopped, they are snapshotted to disk then cost just $0.5/month. They wakeup with the same state (memory and processes too) on ssh connections, or web endpoint activity, or just just a cron schedule. When you dont need the box... Just delete it and stop paying. No subscription, managed via ssh

stop-on-idle + wake-on-HTTP https://github.com/tastyeffectco/sandboxd/

I'm a relatively basic claude code user, basically just running a few instances in different terminal tabs and monitoring them pretty closely, but I could definitely see value in being able to dump a bunch of code and tools into a workspace where there's no credentials present and just set an agent some goals to research or try a bunch of things in a mostly unsupervised manner.

Yeah I have some stuff which is supposed to be “there” for months with the agents continually moving it forward. Not to mention the need to run different software. Running local VMs for now.

Fargate does not use Firecracker, it is simply ec2 instances.

Pretty sure they invented Firecracker for Lambda. Iirc they were previously using a hot pool of EC2 instances behind the scenes with each customer getting their own instances and lambdas sharing capacity on an instance. Firecracker made it possible to spin up VMs in realtime instead of having spare capacity laying around.

That said, Fargate does kind of seem like a superior option

Edit: I guess this supports suspend and fast resume so invocation time should be somewhat better than Fargate.

I don't think Fargate fits for the use case they are describing. If you're running your own (trusted) code, then of course there's no reason to worry about containment threats. But the threat here is that you have to execute arbitrary, untrusted code that is presumptively malicious. It's a very different scenario and requires considerable measures to safeguard properly. You can't have a Fargate Task that runs multiple containers, one for each user, for instance, or even run multiple Fargate Task instances, one for each user, because you're still having them all share a virtual EC2 host (well technically a pool of EC2 servers but it's one hypervisor and shared virtual kernel, essentially) that would be compromised if any one container escapes. If you need true hypervisor-level host kernel isolation on a per-user basis due to the risk of containment, with guest worker microVM threads, plus the whole thing needs to scale and also needs to pause and restore very quickly and keep track of state upon restoration, it's actually a pretty hard challenge to build on AWS with existing tools. The problem arises with any interactive AI agent environment that scales on a per-user basis, for instance, but it also applies to any scenario in which the user needs to execute untrusted arbitrary code on your infrastructure in a sandbox. Fargate isn't the secure choice in that scenario; you would instead use VPC + EC2 + Firecracker + Docker (plus S3 and many others) and use a lot of orchestration scripting and fiddling with load balancers and the like to try to get everything working and scaling. When you combine it with tracking state and also restoring quickly from a paused or suspended state, I can see reasons why this might be the right choice if you want to implement something with an interactive AI agent that isolates at the per-user or per-session layer from the host kernel and is highly secured against containment escape and other vulnerabilities. I'm curious if anyone has used this for the use case described, maybe from AWS? Is this like the AgentCore orchestration that came out maybe like last year?

Also you've been able to run container images in Lambda for some years now.

https://docs.aws.amazon.com/lambda/latest/dg/images-create.h...

Fargate (with Firecracker/Docker) is hellishly complex and forces you into all sorts of AWS BS, first of which is it can't run without some sort of orchestration layer like EKS or ECS.

To deploy on ECS, the simplest option, means that you have to create a private Docker registry, sync base images from docker.io you use, set up IaC, set up a deployer and user identity, create a multiple subnets to allow redundancy, set up health checks, and I didn't even write down the half of it.

If you want a simple enterprise CRUD interal tool, its crazy. And the WORST thing, is that it doesn't have persistent disk, so you either are forced to use slow and expensive EFS or buy into AWS's expensive managed database systems.

And every update to your app goes through a k8s style 'sync image-drain old servers-create new ones-switch over once healthy cycle'. Which has a tendency to fail for mysterious reasons, oh and its undebuggable, and should you notice that an env var is set up incorrectly, you can't just fix it, it means a whole deploy cycle.

I wouldn't wish that stuff on my worst enemy.

In contrast, if you want okay DX, you either go with Lambda, and manage your own EC2s.

Half of AWS's offerings exist to work around the arbitrary limitations they put on their services, as even this thing (as others have correctly noted), comes with this weird 8 hour limit, but even that's far easier to work around (for stuff like running a simple server), than having to deal with the other stuff.

Only EU currently

Cloudflare is cost effective for certain types of workloads, I've heard of businesses getting surprisingly far on the $5/mo worker plan.

This reminds me of Fly.io's model off the top of my head, though its not a self-hosted firecracker as such.

Hetzner is still cheap compared to AWS.

Why do you want to self-host vs. using one of the many providers out there?

Daytona, E2B, OpenComputer, Freestyle, Blaxel, Vercel, Modal, Cloudflare, Tensorlake, Superserve, etc. etc.

Some of them work by pre-purchasing credits, so you can control the blast radius of spend.

Also, if you want a more embedded sandbox runtime as a library instead of a daemon + REST API, you can check out libkrun (and friendly layers on top of it like https://microsandbox.dev/ and https://smolmachines.com/)

For self-hosting, have a look at what we're building with SlicerVM.com (disclosure: I'm the founder). Also runs just as well on Apple Silicon.

We run quite a few Slicer instances on mini PCs and Ryzen builds - also on Hetzner (and yes ouch 120 EUR / mo up to ~ 550 EUR / mo for 16core / 128GB RAM feels almost unfair)

Are you looking for highly ephemeral nodes, where you are writing automation that will use the API to orchestrate it? Or do you just want small microVMs that you launch and kill?

Firecracker just has a ReSTful unix socket with a defined API and launches KVM vms with limited options.

For custom SMB I still think libvirt is a lower entry cost and may have transferable use cases to longer lived VMs, so you can just launch a qemu microvm[0] and use virsh and/or libvirt xml to set up the networking.

The ~400ms boot time of a qemu microvm vs ~120ms for firecracker may not be an issue for some loads, but qemu will also allow you a bit more density of placement than firecracker. qemu microvms will use a bit more memory individually, but they will also tend to use less real system memory with a larger number of microVMs.

It is all tradeoffs, and kata containers are yet another option that may apply depending on your use case.

You can run your own firecracker or qemu/kvm microvms on most instances that allow nested hypervisors, or on a local host. If cost containment is critical to you this is one possible way forward.

Really it just depends on if you want/need ReSTful control, or need to support short lived serverless functions, or if CLIs fit better and you many want to support full VMs.

They both are just Virtual Machine Monitors that targeted different use cases and decided on different tradeoffs.

Just be careful about hosting traditional containers and microVMs on the same system, that config is going to be problematic do to fundamental reasons that are too complex to properly address here.

[0] https://www.qemu.org/docs/master/system/i386/microvm.html

The simplest worthwhile DIY sandbox you can have is to layer two tools: bwrap and gvisor.

    bwrap args -- gvisor args do args -- /path/sandboxee args

bwrap will set up the environment and then gvisor elevates it into a true sandbox.

Standalone gvisor (not the 'do' subcommand) used to be a mess with the OCI json requirement, but recently they began work on presenting their own bwrap interface (likely to pursue AI agent uses) though I wouldn't use it myself yet.

People often look down on gvisor because they think it's some kind of syscall filter, it is not. It can use one of ptrace, seccomp or even KVM to intercept ALL syscalls and service them with it's own logic (which is in Go). Basically it's a VMM and kernel in one.

Not Firecracker, but Incus's system containers seem like a good middle ground between Docker and VMs.

https://linuxcontainers.org/incus/docs/main/explanation/cont...

I just went with qemu and run it in my own machine. It is portable so you run it on other OSes which is handy when everything is under the same desktop app. But I was after better isolation and the ability to be fully in control of the agent environment to pair with local llms. As soon as you lift it to some managed environment it becomes hard to justify all of the necessary steps to manage connections, encryption etc., eg passing credentials for access to other resources.

Do you mind adding https://isorun.ai? We just launched last week. Founder here (Staff SRE with 20 years in Linux, fastest and cheapest SaaS agentic runtime running on heavily modified Firecracker)

Would love to see sandboxd added — it's the self-hosted open-source option with Docker hardening, built-in coding agents (Claude Code, OpenCode), and live preview URLs. https://github.com/tastyeffectco/sandboxd

The problem is that this feature is generally restricted to enterprise customers because VDI systems are such a profitable market. NVIDIA and AMD both only offer this on enterprise cards, and Intel has been very wishy-washy on support in their cards.

If you're looking for a thing to google, look up SR-IOV support on (consumer) GPUs.

Also if you're wondering who the customers of these things tend to be, it's generally the CAD market, law firms, etc. If no one's laptop contains sensitive data and can only stream the desktop of a remote system, the loss or theft of an employee's computer isn't nearly the same kind of a security worry.

This is for people who want both faster execution, and better security isolation for agents/subagents. It is a different use case than yours

I imagine you can have a situation where you let an agent run in a shared env but to access certain tools you spin up a VM just for the tool call duration and then shut it down again. Let’s say you wanna allow the agent to write and run code then you need it to run it somewhere safe

Microvms are better for the VM provider. They use less memory and have a smaller attack surface. Also starting in 100ms means you don't need to add a bunch of async machinery when launching the vms.

in so many cases, docker is more than sufficient for major agent workloads... with no hostile users of course

I dont get it either - I was going to ask the same question but found this.

We have been doing the exact opposite - instead of micro VM's we are giving agents larger VMs.

Previously we were giving them 1GB RAM VM's - now we have upped to 4 GB RAM VM's. When the agent is working - the real cost is in the inference. There is no reason to keep the agent waiting because your VM is too damn slow. So we moved to larger and faster VMs.

The agent might install a package, or run a script - and now it moves along just faster. Not to mention that if the agent is installing a 'fat' SDK, like maybe android sdk, a thicker RAM just moves along everything smoothly without breakages. The incremental amount we pay for the bigger VM is more than justified by the increase in agent performance.

And all the tooling that has already been built up for standard human operated VM's just works pretty well out of the box. We are able to spin up VM's pretty much on demand and purge them clean once the work is done.

We are moving to 8 GB RAMs/4CPUs sometime this year, and GPU's hopefully sometime next.

Yeah, I'm surprised Justin posted this like it was new(s). Wasn't it doing the rounds on the 22nd when it launched?

You can use them for dev environments.

You just have to finish development in 8 hours.

I'm assuming you can launch them again after 8 hours.

They are long-lived if you're a mayfly.

But I think the point is that they should be cheap to set up, and because of the short life, never really contain anything except the potential to compute when needed, not important data.

lambdas are ephemeral on compute, but couldn't you connect up EFS for your long lived data?

then when you launch the next one, its like you are still there?

It just a time limit of the life of a single MicroVM.

Using this for a long lived "developer environment" would be extraordinarily expensive anyhow. Scaling the vCPU + RAM cost of these to the same shape compute optimized Graviton On-Demand EC2 instance (16 vCPU x 32 GB RAM) shows about 4x the cost.

So don't do that. Just use an EC2 instance.

In theory, you could set up a process to move data/filesystem between sessions into and out of s3.

Firecracker doesn't reclaim RAM well, so they put this limit in to make sure you don't suffer from the eventual memory bloat problems.

ramp i think just prebuilts the image/snapshot with the latest checkout version regularly [0]. do you think putting it into the image would address it?

[0] https://builders.ramp.com/post/why-we-built-our-background-a...

Fly.io doesn't set a maximum of 8 hours of alive time on your instance.

Also, MicroVMs can't be exposed directly to the web. Your code running in them can only be executed via API calls with attached auth tokens - so if you wanted to host a public facing API or website with them you'd need to implement your own additional layer in front.

Something I appreciate about Fly (disclaimer: they support my work) is that the pricing is fixed - you pay $1.94/month (less if you suspend your machine) for the smallest instance, up to $976.25/month for the largest (16 CPUs, 128GB) plus predictable costs for volume storage.

The only variable outside your control is bandwidth, and that's unlikely to cause a nasty shock.

Contrast with any of the more "elastic" hosting providers - Vercel, Cloud Run - and you're much less likely to get a horrifying bill if something gets overly-crawled or goes viral.

SaaS offering the usage of LLMs via API. You want to launch something isolated, as quickly as possible, do the minimal amount of work and not have to throwaway all your state.

It's in the very first body paragraph of the article:

"A new class of multi-tenant applications has emerged that all share the need to hand each end user their own dedicated execution environment in which to safely run code that the application developer did not write. AI coding assistants, interactive code environments, data analytics platforms, vulnerability scanners, and game servers that run user-supplied scripts all fit this pattern."

AI agents. Chatbot session of 8 hours is a lot. 16 vCPU might be useful when developing heavy application and agent need run application tests. You can think what infrastructure https://claude.ai/code needs.

> Didn't mean to highjack for self advertisement. > > As the topic matches, .... my project might be appealing to some here

That's exactly what you intended to do. That is the definition of advertising. It is true, many people might like it, so own it. Don't lie about it, even to yourself.

Have you tried using unikraft? I think it might be cheaper imo. Worth a try.

Click on the "MicroVMs" tab of the pricing page: https://aws.amazon.com/lambda/pricing/

You absolutely can run agents on a regular VM. But if you want to build multi-tenant and multi-agent systems with strong security boundaries, then having a VM or MicroVM per agent session (or session with a group of agents) really simplifies things.

When we did AWS AgentCore Runtime last year we introduced session isolation, with MicroVMs per session. You can think of Lambda MicroVMs as the same stack, but generalized to fit a larger number of application patterns.

Isn’t the point that you wanna be able to spin up and down thousands of VM:s on demand (literally a VM just to run a tool and then shut it down until the next tool call)

You don’t have to pay that much. I did pay a couple hundred for a while, but not since I switched to Chinese models along with a $20 ChatGPT subscription.

Also, a single VM is pretty limiting.

I tried this a few days ago. Once you have an image built and ready startup time is fast, but building that original image took 5-10 minutes.

I think it's designed for building an image once and then reusing it many, many times.

Yes. On macOS particularly you can do sandbox-exec(1) with custom / per-task SBPL profiles. Combined with strict control over environment variables that are passed into the agent process plus an outbound firewall like LittleSnitch.

Important is to isolate tasks from each other. Example: for work related tasks I let the agent access Datadog or Docker socket. Everything else does not have access to these.

e2b supports UDP and the pricing structure is different.

i’d say what AWS released looks closer to a bare compute primitive. E2B is up the stack and ships everything around VM like snapshots, networking, integrations.

also, there’s no lock-in, E2B is open-source and can be hosted on any cloud (AWS included).

plus supports bigger boxes, higher concurrency, longer timeouts (24hr).

disclaimer: i work at E2B

Not that I can find in the docs anywhere. Compute only.

No, it doesn’t seem like it.

It is supposed to be a sandbox that you can invoke from agent, langchains of the world, coding agents etc.

check this out https://github.com/smol-machines/smolvm

will have a hosted platform soon with GPU support (vulkan)

Since no one answered, I looked into it. Firecracker you have to provision yourself whereas microVMs are server less and AWS manages all the infra.

Presumably it is Firecracker. It's just a different shape of offering, along with Lambda and Fargate, which are also Firecracker.

The literal first paragraph has a highlighted link that says this runs on Firecracker

It's a product that runs on top of Firecracker.