That helps you if the blast radius is on your machine, but it doesn't really help if the agent is using your credentials to cause some damage with some remote system you have access to.

When I was kicking the tires on pi, one of the first things the agent did was push an update to one of my published Rust crates (not the project it was working on).

That in itself wasn't harmful, but it did convince me it was worth the effort to figure out sandboxing after that.