Yeah, have you tried `mngr` by Imbue? It seems to have a bunch of the features you're looking for.
The networking part I can fix with a second docker container and network_mode I think.
The centralised key and permission management and agent dashboarding is severly lacking though. But that's for now my least worry I think.
Latchkey does support some form of permissions management too.