They don't have arbitrary access over your file system. They ask permission for doing most everything. Even reading files, they can't do that outside of the current working directory without permission.
If you were being really paranoid then I guess they could write a script in the local directory that then runs and accesses other parts of the filesystem.
I've not seen any evidence an agent would just do that randomly (though I suppose they are nondeterministic). In principle maybe a malicious or unlucky prompt found somewhere in the permitted directory could trigger it?