Key Management Service creates and maintains private keys for you and provides an API to easily encrypt or decrypt some data. Basically call method KMS.Decrypt("name_of_key_to_use", <bytes>) and get back decrypted content. Secrets are simple text files encrypted with KMS and stored in a private storage bucket. For example we have something like "database-secrets.dev.json.encrypted".
We have a small library that took a day to write, used in all of our projects that does the following on startup: open private storage bucket, download encrypted file, call the KMS API, decrypt the file, and parse the raw contents as json. Now the app has the secrets in-memory to be used anywhere. No infrastructure required, nothing on disk and this is universally accessible whether inside Google cloud or on local machine. Takes under 1 second when running in the cloud.
I dont think I can do better than the documentation but let me know if you have any questions.
You can limit access to the keys based on IAM instance profiles. So that only certain instances can access specific credentials.
Service accounts are necessary to run anything in GCP anyway but can be used externally (like the gcloud CLI on your desktop) or a similar setup specific to AWS or Azure if that's your primary provider.