I can't seem to understand how a "secrets manager" helps things. Could someone who does ELI5 why it's better than a config file with permissions locked down?
Secrets may still need to go to config files for third party stuff, but you can write your in-house applications to fetch secrets at runtime and hold them only in memory. That's less opportunity for compromise vs. both memory and disk. Also have heard that Linux's process address space isolation is less prone to vulnerabilities than user account separation or filesystem permissions. Not really sure how true that is.
At scale, you can very granularly define policies for each secret. When a secret is accessed, it is done so through a user or application identity. Each access is also logged.
you should only let the instance access the secret it requires.
As OP wrote, you did not solve it, just moved it to a different level.
A typical deployment might involve placing the manager on a secure host that has access to generate and rotate keys, for example.
The manager can then configured to re-generate keys and vend them on-demand to instances that need require them. You can configure these keys with very limited access, and also make them expiring.
The manager then becomes effectively a keystore that can never export master keys, but only vends out some limited-scope keys to other instances.
Other instances would have to authenticate using some pre-configured host keys or even be authenticated directly though the cloud provider.
If your instances are compromised, the worst someone can do is to get access to a limited-scope key that will expire.
Hopefully you have other measures in place that would prevent and detect someone from just sitting on an instance and sucking up all your data and exporting them.