Yes. On macOS particularly you can do sandbox-exec(1) with custom / per-task SBPL profiles. Combined with strict control over environment variables that are passed into the agent process plus an outbound firewall like LittleSnitch.

Important is to isolate tasks from each other. Example: for work related tasks I let the agent access Datadog or Docker socket. Everything else does not have access to these.