most of the pain at scale isn't the agents themselves, it's observability. once you're past a handful you basically need per-agent tracing or you're debugging blind, worth borrowing tooling from how people do this for microservices
If you are using a proprietary harness like CC or Codex then look into automatically ingesting their memory logs and internal datastores to track progress and then instrument the host to send back metrics